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THE  DEVELOPMENT  OF  PERFORMANCE  EVALUATION  SCALES 
FOR  FIRST-LEVEL  ARMY  CIVILIAN  SUPERVISORS 


Introduction 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI)  at  the  request  of  the  Department  of  the  Army  Directorate  for  Civilian  Personnel  has 
undertaken  research  dealing  with  civilian  supervisors  in  the  Army.  The  goals  are  to  evaluate 
the  supervisory  aspects  of  the  job  and  to  develop  a  set  of  supervisory  selection  measures, 
including  a  scoreable  in-basket  exercise,  a  biographical  data  instrument,  and  an  instrument  for 
measuring  temperament.  This  region  describes  the  development  of  a  set  of  rating  scales  that 
will  be  used  to  provide  a  criterion  against  which  the  scoreable  in-basket  exercise  and  the 
biographical  data  and  temperament  instruments  will  be  validated. 

Because  validation  efforts  require  rating  scales  of  high  quality,  it  was  decided  to 
develop  behaviorally  anchored  rating  scales.  There  are  several  advantages  to  using 
behaviorally  anchored  scales  as  opposed  to  other  types  of  rating  scales  (Campbell,  Dunnette, 
Arvey,  and  Hellervik,  1973).  Behaviorally  anchored  rating  scales  emphasize  objective 
observation  rather  than  subjective  evaluation.  They  are  constructed  to  reflect  performance 
requirements  regarded  as  important  by  those  knowledgeable  about  the  job.  Behaviorally 
anchored  rating  scales  define  the  relevant  and  important  performance  requirements  in  concrete 
terms.  In  the  procedure  used  to  develop  the  scales,  job  experts  agree  on  the  effectiveness 
levels  of  scaled  job  behaviors  used  as  performance  effectiveness  "anchors."  In  sum,  raters 
can  compare  the  observed  performance  of  a  supervisor  to  behavioral  benchmarks  or  standards 
of  effectiveness,  resulting  in  more  objective  performance  judgments. 

An  inductive  behavioral  analysis  strategy  can  be  used  to  develop  and  evaluate 
behaviorally  based  rating  scales  (Campbell  et  al.,  1973).  This  process  requires  individuals 
familiar  with  a  job’s  performance  demands  to  generate  examples  of  behavior  observed  on  that 
job.  The  examples  can  represent  a  range  of  performance  effectiveness  from  high  to  low.  The 
behavioral  examples,  or  critical  incidents,  are  then  content  analyzed  to  form  dimensions  or 
categories  of  supervisory  effectiveness  and  submitted  to  a  retranslation  procedure.  During  the 
retranslation  procedure,  experienced  job  incumbents  evaluate  each  example,  place  it  in  a 
category,  and  rate  the  level  of  effectiveness  it  reflects.  Examples  of  behavior  for  which  there 
is  good  agreement  in  the  retranslation  procedure  are  used  to  form  behavioral  statements 
anchoring  different  levels  of  effectiveness  on  each  of  the  dimensions.  The  dimensions  and 
behavioral  anchors  comprise  the  supervisory  rating  scales.  This  procedure  was  used  in 
developing  the  Army-Wide  Rating  Scales  used  in  Project  A:  Improving  the  Selection, 
Classification,  and  Utilization  of  Army  Enlisted  Personnel  (Pulakos  &  Borman,  1985). 

This  report  documents  the  process  by  which  the  set  of  first-level  supervisor 
performance  evaluation  rating  scales  was  developed.  It  describes  in  detail  the  critical  incident 
methodology  employed  to  develop  the  scales,  and  preliminary  psychometric  analyses  of  scale 
data  collected  in  workshops  that  were  held  to  try  out  the  pre-testing  validation  procedures. 
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Method 


A  six-step  procedure  was  used  to  develop  the  rating  scales.  The  steps  consisted  of: 

1.  A  brief  examination  of  some  dimensions  of  supervisory  performance  found  in 
earlier  research. 

2.  The  conduct  of  a  series  of  workshops  where  participants  wrote  critical  incidents 
that  exemplified  good  and  poor  supervisory  performance. 

3.  Analysis  of  the  critical  incident  workshop  data  from  which  an  intermediate  set 
of  performance  dimensions  emerged. 

4.  A  series  of  retranslation  workshops  in  which  participants  independently 
assigned  the  critical  incidents  to  the  performance  dimensions.  The  workshop 
participants  also  evaluated  the  intermediate  performance  dimensions.  During 
the  last  retranslation  workshop,  the  participants  also  tried  out  a  set  of  scales 
based  on  the  intermediate  dimensions. 

5.  The  construction  of  the  trial  performance  evaluation  scales  based  on  the 
analysis  of  the  retranslation  workshop  data. 

6.  The  administration  of  the  trial  scales  during  a  pilot  test  of  the  selection 
instruments  and  associated  administrative  procedures. 

These  steps  are  described  in  detail  below. 


Examination  of  Supervisory  Performance  Dimensions 

A  brief  examination  was  made  of  the  dimensions  of  supervisory  performance  found  in 
other  studies.  The  primary  purpose  of  this  review  was  to  familiarize  project  personnel  with 
the  kinds  of  categories  of  supervisory  performance  that  other  researchers  and  practitioners 
have  formulated.  Although  it  was  planned  that  the  content  of  the  critical  incidents  would 
determine  the  performance  categories  for  which  scales  would  be  developed,  it  was  felt  that 
awareness  of  previously  derived  categorization  schemes  would  be  of  value  to  those 
responsible  for  the  content  analysis  of  the  critical  incidents.  Such  awareness  would  help 
sensitize  the  content  analysts  to  different  areas  of  supervisory  performance  and  also  to  foster 
agreement  among  them. 

The  examination  concentrated  on  the  responsibilities  and  tasks  of  first-level  civilian 
supervisors  working  for  the  federal  government,  especially  those  working  for  the  Department 
of  Defense  and  the  Army.  Given  this  concentration,  three  primary  sources  of  information 
were  reviewed  in  detail:  (1)  The  Supervisory  Grade  Evaluation  Guide,  published  by  the  U.  S. 
Civil  Service  Commission  in  1976,  (2)  the  Office  of  Personnel  Management  Job  Grading 
Standard  for  Supervisors,  and  (3)  the  recent  job  analysis  of  Army  civilian  first-level 
supervisors  conducted  by  Rosenthal,  Riegelhaupt,  and  Ziemak  (1988).  The  first  two 
references  provided  general  information  about  the  jobs  of  federal  civilian  supervisors;  the 
latter  provided  specific  information  about  the  jobs  of  Army  civilian  first- level  supervisors. 

According  to  the  standards  of  the  Supervisory  Grade  Evaluation  Guide,  published  by 
the  U.  S.  Civil  Service  Commission  in  1976,  supervisors  are  responsible  for: 
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1.  Ensuring  the  timeliness,  amount,  and  quality  of  subordinates’  work 

2.  Reviewing  subordinates’  products 

3.  Planning  subordinates’  work  and  setting  their  schedules  and  priorities 

4.  Assigning  work  to  subordinates  based  on  priorities,  task  difficulty,  and 
employee  capabilities 

5.  Evaluating  subordinate  performance 

6.  Advising  and  counseling  employees  on  work  and  administrative  matters 

7.  Selecting  and  recommending  candidates  for  positions  within  the  unit  and  for 
reassignments  and  promotions 

8.  Resolving  employee  complaints  and  grievances  and  referring  more  serious 
matters  to  proper  levels 

9.  Warning  and  reprimanding  employees  and  recommending  further  disciplinary 
actions  as  required 

10.  Identifying  developmental  and  training  needs  of  subordinates  and  making 
appropriate  provisions  for  them. 

In  the  Office  of  Personnel  Management  classification  guide  for  Wage  Grade 
supervisors  (Job  Grading  Standard  for  Supervisors),  FPM  Supplement  512-1,  August  1982) 
first-level  supervisors  are  defined  as  individuals  who  supervise  individuals,  who,  in  turn,  do 
not  supervise  anyone.  Supervisory  responsibilities  of  first-level  supervisors  are  given  as: 

1.  Planning:  setting  priorities,  assigning  work,  establishing  deadlines,  considering 
the  number  and  skill  of  available  workers. 

2.  Work  Direction:  motivating  workers,  defining  standards,  explaining 
assignments,  and  coordinating  unit’s  progress  with  other  units. 

3.  Personnel  Administration:  controlling  leave,  conducting  performance 
appraisals,  adjusting  grievances,  and  taking  disciplinary  actions. 

A  much  more  refined  list  of  supervisory  tasks  may  be  found  in  the  report  prepared  by 
Rosenthal  et  al.  (1988).  They  studied  lists  of  performance  categories  and  tasks  in  a  wide 
range  of  studies  including  Hill,  Kerr,  and  Broedling  (1984),  McAreavy,  King,  and  Eichhom 
(1985),  Bass  (1981),  Corts  (1982),  Fleishman  (1953,  1973),  Fleishman  and  Hunt  (1973),  OPM 
(1985),  Gibb  (1969),  Hollander  and  Julian  (1969),  Stodgill  (1974),  Vroom  (1976),  and  Yukl 
(1981).  From  this  literature,  and  from  descriptions  of  supervisory  jobs  in  various  Army  and 
other  federal  agency  documents  and  reports,  as  well  as  from  interviews  with  incumbent  Army 
civilian  first-level  supervisors,  Rosenthal  et  al.  (1988)  derived  a  preliminary  list  of  supervisory 
tasks. 


The  preliminary  list  of  tasks  was  revised  in  a  series  of  68  workshops  conducted  at  18 
Army  installations.  The  workshops  were  attended  by  427  Army  civilian  first-level  supervisors 
who  were  asked  to  review  the  task  list  (which  was  modified  after  each  successive  workshop) 
and  to  make  any  revisions,  additions,  or  deletions  that  would  enable  the  list  to  more 
accurately  reflect  their  supervisory  positions. 

A  total  of  226  tasks  emerged  from  this  process.  These  tasks  were  sorted  into  1 1 
categories: 
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1.  Personnel  staffing 

2.  Position  management 

3.  Equal  Employment  Opportunity 

4.  Managing  the  workforce 

5.  Training  and  development 

6.  Employee-management  communications 

7.  Administrative  duties 

8.  Safety 

9.  Managing  work  and  operations 

10.  Planning/budgeting/executing 

11.  Security  concerns 

The  task  list  was  distributed  to  4,400  Army  civilian  first-level  supervisors  working  at 
155  locations  world-wide.  These  supervisors  were  asked  to  indicate  the  relative  amount  of 
time  they  spend  on  the  tasks  and  to  rate  the  relative  importance  of  the  tasks  for  their  jobs. 
Rosenthal  et  al.  (1988)  present  a  table  showing  the  39  tasks  on  which  2,764  responding 
supervisors  on  the  average  indicated  they  spent  the  most  time  (pp.  22-23).  Another  table 
gives  the  39  tasks  that  they  rated  as  most  important  to  their  jobs  (pp.  24-25). 

The  tasks  listed  in  the  two  tables  cited  in  Rosenthal  et  al.  (1988)  were  studied 
carefully  along  with  other  performance  categorization  schemes  by  the  HumRRO  psychologists 
who  were  going  to  be  responsible  for  the  later  content  analysis  of  the  critical  incident 
workshop  data.  Based  upon  the  examined  performance  categorization  schemes,  a  preliminary 
set  of  25  supervisor  performance  factors  was  developed.  The  plan  wa'  to  ask  participants  in 
the  critical  incident  workshops  to  evaluate  the  performance  factors  in  terms  of  their  relative 
importance,  rating  difficulty,  appropriateness,  and  content  overlap.  However,  time  did  not 
permit  most  workshop  participants  to  complete  the  evaluations,  and  the  data  that  were 
collected  were  not  analyzed.  The  list  of  preliminary  performance  factors  and  the 
accompanying  evaluation  forms  are  given  in  Appendix  A. 


The  Critical  Incident  Workshops 

A  series  of  seven  four- hour  critical  incident  workshops  were  held  in  Spring  1989  at 
various  Army  facilities  that  employ  civilian  workers.  A  total  of  93  first-  and  second-level 
civilian  supervisors  attended  the  workshops  in  groups  ranging  in  size  from  6  to  19 
supervisors.  The  workshop  participants  were  first  given  an  orientation  to  the  research  project 
by  the  workshop  leader.  The  orientation  emphasized  the  need  to  develop  and  validate  civilian 
first-level  supervisory  selection  instruments.  (All  materials  used  in  the  critical  incident 
workshops  are  given  in  Appendix  A.)  An  overview  of  the  workshop  activities  was  then 
presented.  The  overview  stressed  the  importance  of  developing  a  rating  form  that  would 
allow  raters  to  provide  accurate  assessments  of  Army  civilian  supervisors’  performance  and 
effectiveness. 

Different  kinds  of  rating  forms  were  discussed  next.  The  advantages  of  behavior- 
based  rating  scales,  e.g.,  the  emphasis  on  objective  observation  rather  than  subjective 
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evaluation,  were  listed.  The  role  that  specific  incidents  describing  effective,  ineffective,  or 
average  performance  played  in  the  development  of  the  behavior-based  scales  was  explained. 

Before  writing  any  incidents,  participants  were  given  a  short  training  session  on  how 
to  write  useable  incidents.  They  were  told  to  emphasize  what  the  individual  first-level 
supervisor  actually  did  that  made  the  workshop  participant  feel  that  the  supervisor  was 
effective  or  ineffective.  They  were  asked  to  describe  clearly  and  concisely  the  background 
and  consequences  of  the  incident.  They  were  then  given  three  poorly  written  incidents  am. 
were  told  what  was  wrong  with  them  and  shown  how  the  incidents  could  have  been  written 
correctly. 

The  workshop  participants  then  spent  about  2  1/2  hours,  including  a  10  minute  break, 
writing  critical  incidents.  The  workshop  leader  answered  questions  and  spot-checked  the 
incidents  as  they  were  being  written.  Altogether,  the  93  workshop  participants  produced  841 
incidents  or  about  an  average  of  9  incidents  per  participant. 

If  there  was  time  available  the  workshop  leader  led  a  short  informal  discussion  on 
problems  involved  in  measuring  first-level  supervisor  performance.  A  set  of  discussion 
questions  was  provided  the  participants,  but  no  attempt  was  made  to  address  these  questions 
rigorously.  The  main  purpose  of  the  session  was  to  give  the  participants  a  chance  to  express 
their  feelings,  thoughts,  and  ideas  about  the  workshops  and  performance  measurement  issues. 


Analyses  of  the  Critical  Incident  Workshop  Data 

The  primary  objective  of  the  analysis  of  the  critical  incidents  was  to  formulate  a  set  of 
first-level  supervisor  performance  categories  into  which  most,  if  not  all,  of  the  incidents  could 
be  reliably  placed.  That  is,  we  wanted  the  performance  categories  developed  to  meet  the 
criterion  that  knowledgeable  observers  of  the  given  incidents  would  independently  say  that  the 
supervisory  behavior  displayed  in  the  incidents  exemplified  the  same  basic  supervisory 
performance  factor  or  dimension.  The  rationale  for  this  criterion  is  that  if  observers  could 
agree  that  specific  types  of  behaviors  fell  into  certain  categories  of  performance,  then  they 
would  be  more  apt  to  agree  on  their  ratings  of  individual  supervisors  on  behaviorally 
anchored  scales  used  to  rate  performance  in  the  separate  categories.  Also,  if  observers  could 
agree  that  sizeable  numbers  of  critical  incidents  fell  into  several  specific  categories,  then  the 
number  of  incidents  so  categorized  and  the  content  of  those  incidents  could  be  used  to  help 
determine  which  aspects  of  supervisory  performance  should  be  captured  by  the  rating  scales 
and  what  the  content  of  the  behavioral  anchors  of  those  scales  should  be.  The  procedures 
used  to  achieve  this  objective  are  described  below. 

The  critical  incidents  obtained  in  the  workshops  were  divided  into  three  subsets.  The 
first  subset  consisted  of  the  approximately  350  incidents  obtained  in  the  first  three  workshops. 
The  second  and  third  subsets,  consisting  of  approximately  250  incidents  each,  were  obtained 
from  the  fourth/fifth  and  sixth/seventh  workshops,  respectively. 

An  iterative  procedure  which  involved  successive  content  analyses  of  the  three  critical 
incident  subsets,  was  used  to  formulate  the  performance  categories.  The  contents  of  the 
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incidents  in  the  first  subset  were  examined  independently  by  three  HumRRO  psychologists 
whose  task  was  to  formulate  and  name  a  set  of  performance  categories  into  which  all  the 
incidents  could  be  placed.  After  they  had  separately  developed  the  performance  categories, 
the  three  analysts  met  and  compared  their  category  names,  as  well  as  their  placement  of 
incidents  in  the  categories.  Similarities  and  differences  in  the  names  and  content  of  the 
categories  were  discussed  in  detail.  A  set  of  initial  performance  categories  upon  which  the 
three  analysts  could  agree  were  defined  at  the  conclusion  of  the  discussions. 

These  initial  performance  categories  were  then  used  by  the  three  analysts  to  categorize 
the  critical  incidents  in  the  second  subset  of  incidents.  Again  the  analysts  worked 
independently.  They  noted  whether  any  incidents  did  not  seem  to  fall  into  any  of  the  existing 
categories.  They  noted  which  categories  they  had  the  most  difficulty  with  and  how  changes 
in  the  number  and  definition  of  the  categories  might  improve  their  ability  to  place  the 
incidents. 

The  analysts  then  met  and  discussed  the  suggested  category  changes.  They  compared 
their  placements  of  the  incidents  in  the  categories,  noted  which  categories  had  the  least 
amount  of  overlap  in  incident  placement,  and  tried  to  determine  the  reason  for  the  placement 
discrepancies.  They  examined  the  specific  incidents  that  they  had  each  placed  in  a  different 
category  and  tried  to  reconcile  their  placements,  changing  the  category  definitions  as  deemed 
necessary  to  achieve  greater  agreement. 

The  revised  set  of  performance  categories  was  then  applied  in  the  categorization  of  the 
critical  incidents  comprising  the  third  subset  of  incidents.  The  same  procedure  used  for  the 
second  subset  of  incidents  was  used.  The  analysts  first  independently  placed  the  incidents  in 
the  categories.  Difficulties  encountered  by  the  analysts  in  categorizing  the  incidents  were 
noted.  The  analysts  then  met  and  compared  their  placements.  The  amount  of  agreement  they 
had  by  category  was  determined.  The  performance  categories  were  then  revised  again. 

Twelve  performance  categories  emerged  from  this  iterative  procedure.  These 
categories  were  used  in  the  retranslation  workshops.  The  category  definitions  are  given  in 
Figure  1. 

A  set  of  12  draft  first-level  supervisor  ratings  scales  was  also  developed  based  on  the 
category  definitions.  These  draft  scales  were  used  to  get  a  very  preliminary  look  at  some  of 
the  likely  psychometric  characteristics  of  the  scales.  Through  using  these  draft  scales,  the 
authors  hoped  to  find  out  whether  something  was  vitally  wrong  with  the  scales  from  a 
measurement  point  of  view.  The  scale  contents,  that  is,  the  behavioral  descriptors,  were 
based  on  the  critical  incidents  that  the  content  analysts  agreed  belonged  in  the  given 
performance  categories.  The  draft  scales  are  presented  in  Appendix  B,  which  also  contains 
copies  of  the  materials  used  in  the  retranslation  workshops. 

Retranslation  Workshop  Procedures 

Although  841  incidents  were  collected  at  the  critical  incident  workshops,  not  all  of 
them  could  be  used  in  the  retranslation  workshops.  Some  were  simply  illegible.  Others  were 
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A.  Enforcing  Standards  of  Behavior  and  Adherence  to  Pc  licies/Regulations 

•  Sets  clear  standards  of  acceptable  behavior  on  the  job.  Insists  employees  meet 
these  standards  and  follow  applicable  policy  directives,  rules  and  regulations. 
Takes  appropriate  unofficial  and  official  disciplinary  actions  to  correct  failings. 

versus 

•  Ignores  such  behaviors  as  chronic  tardiness,  long  lunches,  improper  dress, 
excessive  phone  calls.  Is  careless  concerning  security  matters,  record  keeping, 
property  disposition,  travel  orders,  and  the  like.  Over  or  under  reacts  in 
administering  discipline,  failing  to  obtain  improvement. 


B.  Maintaining  Employee  Safety,  Health  and  Physical  Well  Being 

•  Takes  active  steps  to  minimize  hazards  in  the  work  environment.  Calls  attention 
of  subordinates  and  property  authorities  to  safety  and  health  problems.  Ensures 
that  unit  work  environment  is  clean,  adequately  heated  and  lighted,  free  from 
clutter,  noisy  distractions,  cigarette  smoke,  and  the  like. 

versus 

•  Ignores  safety  and  health  hazards.  Fails  to  enforce  safety  and  health  related  rules, 
e.g.,  smoking  restrictions.  Shows  no  concern  for  physical  well  being  of 
subordinates. 


C.  Completing  Formal  Employee  Performance  Appraisals 

•  Completes  performance  appraisals  for  annual  reviews  and  promotions  on  time, 
and  constructively.  Discusses  strength  and  weaknesses  of  employees.  Evaluates 
performance  objectively  and  fairly. 

versus 

•  Delays  completing  performance  appraisals;  fails  to  indicate  how  performance 
could  be  improved.  Plays  favorites,  gives  high  (or  low)  ratings  to  individuals 
who  don’t  deserve  them. 


Figure  1.  First-Level  Supervisor  Performance  Categories 
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D.  Resolving  Employee  Conflicts 

•  Actively  intercedes  to  reduce  conflicts  among  employees.  Listens  to  each  side 
attentively  and  tries  to  promote  positive  interactions.  Takes  strong  action  when 
necessary  to  reduce  tensions  and  conflicts. 

versus 

•  Ignores  conflicts  among  employees,  lets  them  get  out  of  hand;  jumps  in 
thoughtlessly,  without  ascertaining  facts.  Shows  bias  in  handling  disputes. 


E.  Providing  Personal  and  Career  Counseling 

•  Provides  confidents  advice  and  help  to  subordinates  whose  personal  problems 
are  interfering  with  performance.  Advises  subordinates  on  job  growth  and  career 
possibilities  and  how  to  overcome  personal  obstacles  to  job  success. 

versus 

•  Ignores  personal  problems  of  subordinates  or  makes  them  worse  through 
callousness  and  insensitivity.  Discourages  career  and  personal  growth  of 
subordinates. 


F.  Communicating  with  Subordinates 

•  Keeps  subordinates  informed  about  impending  personnel  and  work  load  changes, 
installation  activities,  new  policies,  and  other  matters  of  concern  and  interest  to 
them.  Regularly  holds  meetings  with  subordinates  to  discuss  such  items. 
Encourages  employee  suggestions  and  comments. 

versus 

•  Fails  to  tell  employees  about  upcoming  changes  that  may  impact  their  jobs  and 
work  environment  Keeps  such  information  to  self,  sometimes  to  detriment  of 
employees’  job  performance  and  satisfaction.  Discourages  exchange  of  ideas 
and  opinions  on  job  related  matters. 


Figure  1.  First-Level  Supervisor  Performance  Categories  (Continued) 
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G.  Assuring  Technical  Competence  of  Subordinates 

•  Makes  sure  that  subordinates  understand  how  to  do  their  tasks  properly  by 
providing  relevant  and  timely  technical  guidance  and  feedback.  Closely 
monitors  quality  and  timeliness  of  subordinates’  products  and  provides 
constructive  suggestions  for  improving  performance.  Provides  any  needed  on- 
the-job  training  or  ensures  that  it  is  obtained  elsewhere. 

versus 

•  Doesn’t  provide  the  technical  guidance  or  training  necessary  for  subordinates  to 
do  high  quality  work.  Fails  to  monitor  subordinates’  task  accomplishment  and 
to  ensure  timely,  accurate,  or  complete  reports,  quality  products. 


H.  Showing  Initiative,  Extra  Effort,  and  Composure  Under  Stress 

•  Reacts  to  job  demands  and  stressful  conditions  with  initiative,  extra  effort,  and 
composure.  Voluntarily  tackles  new  tasks.  Swiftly  acts  to  remove  unexpected 
barriers  to  job  success.  Efficiently  accomplishes  complex  tasks  under  extreme 
time  pressures. 

versus 

•  Seems  to  fall  apart  under  pressure,  not  able  to  rise  to  the  occasion.  Doesn’t  do 
more  than  what’s  minimally  necessary  to  maintain  position.  Fails  to  put  in  extra 
effort  when  needed. 


I.  Showing  Integrity  and  Honesty 

•  Is  honest  and  straightforward  in  dealings  with  others.  Follows  same  rules  set  for 
subordinates.  Gives  subordinates  credit  when  credit  is  due.  Sets  a  role  model 
for  subordinates  to  follow. 

versus 

•  Exploits  position  for  personal  gain.  Takes  sole  credit  for  subordinates’  work. 
Lies  to  or  misleads  subordinates.  Engages  in  dishonest  acts  such  as  pilfering, 
falsifying  leave  records,  charging  time  not  worked. 


Figure  1.  First-Level  Supervisor  Performance  Categories  (Continued) 
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J.  Maintaining  Employee  Morale  and  Motivation 

•  Takes  steps  to  ensure  that  employee  moral  is  high  and  that  employees  are  well 
motivated.  For  example,  provides  awards  fairly,  pays  individualized  attention  to 
employees,  arranges  special  ceremonies  and  events,  encourages  team  spirit  and 
high  unit  performance. 

versus 

•  Makes  employees  feel  generally  unappreciated.  Fails  to  consider  special  needs 
of  subordinates  or  back  them  up  when  they  need  support.  Frequently  undercuts 
subordinates. 


K.  Assuring  Unit  Functioning 

•  Carefully  plans  and  schedules  unit  work.  Reorganizes  unit  activities  and  staff 
responsibilities  and  annual  leave  to  meet  changes  in  work  loads  and  deadlines. 
Assigns  employees  to  achieve  fair,  challenging  work  load  distribution.  Makes 
sure  subordinates  are  in  appropriate  grades.  Protects  unit  from  outside 
interference,  staff  reductions,  mission  loss.  Develops  own  professional  skills  as 
a  manager. 


versus 

•  Plans  and  schedules  inadequately  so  that  backlogs,  bottlenecks,  machine  failures, 
supply  and  staff  shortages  occur.  Delegates  work  tasks  unequally;  assigns  work 
without  considering  duties  and  capabilities  of  staff.  Allows  major  disruption  of 
unit  functioning  to  occur.  Makes  no  attempt  to  grow  as  a  supervisor  or 
manager. 


L.  Communicating,  Coordinating  With  Supervisors  and  Other  Units 

•  Communicates  effectively  with  supervisors  and  other  units.  Keeps  them 
informed  about  unit  activities,  product  delivery,  personnel  availability,  and 
problems.  Handles  disagreements  with  tact,  diplomacy,  and  persuasiveness. 
Coordinates  inter-unit  activities  to  improve  mutual  productivity. 

versus 

•  Gets  into  no-win  struggles  with  supervisors  and  other  units.  Is  rude,  abrasive, 
and  disrespectful  to  them.  Hides  problems  until  it’s  too  late  to  effectively  solve 
them.  Fosters  "them"  versus  "us"  feelings. 


Figure  1.  First-Level  Supervisor  Performance  Categories 
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too  fragmentary  to  constitute  an  integral,  coherent  incident.  Other  incidents  were  too 
complex,  involving  several  distinctly  different  aspects  of  supervisory  performance.  Still 
others  essentially  described  the  behavior  of  second-level  supervisors  interacting  with 
subordinate  supervisors.  And  other  incidents  were  practically  replicates  of  each  other,  and 
might  even  have  been  describing  the  same  supervisor’s  behavior. 

Not  only  were  many  incidents  unusable,  but  difficulties  in  making  arrangements  to 
obtain  large  numbers  of  participants  for  retranslation  workshops  led  to  the  conclusion  that  the 
number  of  critical  incidents  used  in  the  retranslation  process  would  have  to  be  curtailed.  The 
number  of  incidents  was  subsequently  reduced  to  600  mostly  through  the  elimination  of 
incidents  having  one  or  more  of  the  aforementioned  problems.  The  remaining  incidents  were 
edited  minimally  (to  eliminate  spelling  and  egregious  grammatical  errors)  and  typed  out  on 
separate  sheets  for  legibility  purposes. 

As  600  incidents  were  far  too  many  for  any  one  judge  to  retranslate  in  a  four  hour 
workshop,  it  was  decided  to  split  the  incidents  up  among  the  workshop  participants.  (Based 
on  experience  in  Project  A  retranslation  workshops,  we  figured  it  would  take  between  one  and 
one  and  a  half  minutes  on  the  average  to  judge  each  incident.)  The  overall  plan  for  the 
retranslation  workshops  railed  for  obtaining  at  least  60  participants  who  would  each 
retranslate  160  of  the  600  critical  incidents.  This  would  result  in  at  least  16  participants 
retranslating  each  incident. 

The  incidents  were  presented  to  the  workshop  participants  in  counterbalanced  order 
to  help  assure  that  each  incident  would  be  judged  by  approximately  the  same  number  of 
participants  in  case  participants  were  unable  to  complete  all  of  their  assigned  incidents.  The 
600  incidents  were  randomly  divided  into  30  packets,  each  containing  20  incidents.  The  60 
participants  were  divided  into  three  groups  of  20  each.  Each  group  was  assigned  10  of  the  30 
packets  (i.e.,  200  of  the  600  incidents);  each  participant  received  eight  of  the  10  packets 
assigned  to  the  group.  The  assignment  order  of  the  10  packets  to  the  20  participants  is  shown 
in  Figure  2.  The  remaining  20  packets  were  to  be  similarly  assigned  to  the  remaining  two 
groups  of  20  participants. 

It  was  anticipated  that  some  of  the  workshop  participants  would  complete  the 
retranslation  task  with  sufficient  time  remaining  to  assist  in  the  further  evaluation  of  the 
performance  factors.  A  set  of  evaluation  forms  was  developed.  These  forms  called  for  the 
evaluation  of  the  12  performance  factors  used  in  the  retranslation  process  on  four  criteria: 
importance,  ease  of  rating,  applicability  across  different  types  of  jobs,  and  redundancy. 

Appendix  B  contains  a  copy  of  materials  and  instructions  used  in  the  retranslation 
workshops.  The  introductory  materials  used  were  essentially  the  same  as  the  materials  used 
in  the  critical  incident  workshops.  After  studying  the  definitions  of  the  12  performance 
categories  (see  Figure  1),  the  workshop  participants  examined  each  incident  in  the  packages  in 
turn.  Once  they  had  decided  in  which  performance  category  an  incident  belonged,  they  rated 
the  effectiveness  of  the  described  behavior  on  a  nine-point  scale,  where  "9"  was  extremely 
effective  and  "1"  was  extremely  ineffective.  They  then  proceeded  to  the  next  incident. 
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Participant 

Packet  ID  Number 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

01 

1* 

2 

3 

4 

5 

6 

7 

8 

02 

1 

2 

3 

4 

5 

6 

7 

8 

03 

1 

2 

3 

4 

5 

6 

7 

8 

04 

8 

1 

2 

3 

4 

5 

6 

7 

05 

7 

8 

1 

2 

3 

4 

5 

6 

06 

6 

7 

8 

1 

2 

3 

4 

5 

07 

5 

6 

7 

8 

1 

2 

3 

4 

08 

4 

5 

6 

7 

8 

1 

2 

3 

09 

3 

4 

5 

6 

7 

8 

1 

2 

10 

2 

3 

4 

5 

6 

7 

8 

1 

11 

8 

7 

6 

5 

4 

3 

2 

1 

12 

8 

7 

6 

5 

4 

3 

2 

1 

13 

8 

7 

6 

5 

4 

3 

2 

1 

14 

7 

6 

5 

4 

3 

2 

1 

8 

15 

6 

5 

4 

3 

2 

1 

8 

7 

16 

5 

4 

3 

2 

1 

8 

7 

6 

17 

4 

3 

2 

1 

8 

7 

6 

5 

18 

3 

2 

1 

8 

7 

6 

5 

4 

19 

2 

1 

8 

7 

6 

5 

4 

3 

20 

1 

8 

7 

6 

5 

4 

3 

2 

*  Numbers  in  Table  indicate  order  in  which  participants  retranslated  the  8  packets  of  critical  incidents 

they  were  each 

assigned. 

Figure  2.  Assignment  Order  of  Packets  to  Retranslation  Workshop  Participants 

Five  workshops  were  held  at  various  Army  sites  or  installations  in  the  United  States. 
Fifteen  supervisors  were  requested  to  attend  each  workshop.  Altogether,  the  workshops  were 
attended  by  66  Army  civilian  supervisors.  However,  some  of  the  supervisors  did  not 
complete  retranslating  one  or  two  packets  of  critical  incidents.  Therefore,  during  the  fifth 
workshop,  the  packet  assignment  was  rearranged  to  allow  all  packets  to  be  retranslated  by  at 
least  16  judges.  During  this  last  workshop,  the  12  participating  supervisors  were  asked  to 
retranslate  only  6  packets.  When  they  had  completed  this  retranslation  task,  each  participant 
was  asked  to  rate  the  performance  of  three  supervisors  on  the  12  draft  rating  scales  that  had 
been  constructed  earlier.  The  participants  were  instructed  to  select  the  three  supervisors  they 
believed  most  qualified  to  rate,  but  not  to  identify  these  supervisors  except  by  their  initials 
(which  could  be  fictitious).  It  was  explained  that  the  purpose  of  the  ratings  was  to  try  out  the 
scales  and  not  to  obtain  ratings  of  particular  individuals.  As  the  participants  did  not  identify 
themselves  on  the  rating  forms,  the  anonymity  of  the  ratees  was  assured  by  this  procedure. 
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After  completing  the  ratings  of  the  three  supervisors,  the  participants  in  the  final 
workshop  completed  the  same  four  performance  factor  evaluation  forms  that  the  participants 
in  the  earlier  four  workshops  had  completed  after  retranslating  the  8  packets  of  critical 
incidents  assigned  to  them. 


Construction  of  the  Trial  Supervisor  Performance  Evaluation  Scales 

The  trial  set  of  supervisor  performance  scales  was  developed  from  the  set  of  twelve 
draft  first-level  supervisor  ratings  scales  used  in  the  last  retranslation  workshop.  The 
modification  of  the  draft  scales  was  accomplished  in  five  basic  steps.  First,  the  critical 
incidents  that  had  been  reliably  placed  in  the  various  performance  categories  by  the  workshop 
participants  were  identified.  Second,  the  incidents  were  designated  as  reflecting  high, 
medium,  or  low  performance  levels  based  upon  the  average  effectiveness  level  assigned  the 
incidents  by  the  participants.  Incidents  that  were  assigned  a  wide  range  of  effectiveness 
levels  by  the  participants  were  dropped.  Third,  the  remaining  incidents  were  examined 
carefully  in  conjunction  with  the  corresponding  performance  level  descriptions  given  in  the 
preliminary  scales.  The  scales  were  revised  to  reflect  the  content  of  the  incidents  in  each 
performance  category/level  combination.  Fourth,  the  evaluative  data  collected  on  the 
preliminary  performance  categories  and  scales  were  examined  and  decisions  were  made  as  to 
whether  some  of  the  scales  should  be  further  revised  or  even  dropped.  Finally,  on  the  basis 
of  comments  received  from  reviewers  of  the  scales,  the  performance  content  of  one  of  the 
scales  was  subdivided  into  two  new  scales.  These  five  steps  are  described  further  in  the 
sections  below. 

Identification  of  Relevant  Incidents  bv  Performance  Category 

The  critical  incidents  that  were  placed  in  the  same  supervisor  performance  category  by 
at  least  50%  of  the  66  workshop  participants  were  initially  identified. 

A  check  was  then  made  to  determine  whether  some  of  the  66  workshop  participants 
may  have  produced  rather  atypical  judgments  in  the  retranslation  process.  A  score 
representing  the  amount  of  agreement  with  the  other  judges  was  calculated  for  each 
participant.  This  score  was  the  percentage  of  time  each  judge  had  placed  the  incidents  he/she 
retranslated  into  the  same  performance  categories  that  50%  or  more  of  the  judges  had.  (Only 
the  incidents  that  were  placed  in  the  same  category  by  at  least  50%  of  the  participants  were 
used  in  obtaining  this  score.) 

The  participants’  percentage  agreements  with  the  majority  placement  ranged  from  a 
low  of  53%  to  a  high  of  100%  with  a  mean  of  84%.  Twenty-four  of  the  66  judges  had 
percentage  agreements  of  90%  or  higher.  On  the  basis  of  the  agreement  distribution  obtained, 
it  was  decided  to  drop  two  judges,  one  with  53%  agreement,  the  other  with  56%  agreement. 
These  agreement  indexes  were  more  than  two  and  one  half  standard  deviations  below  the 
mean  agreement  index  for  the  66  participants. 

The  critical  incidents  that  were  placed  in  the  same  supervisor  performance  category  by 
at  least  50%  of  the  remaining  64  workshop  participants  were  then  identified.  The  number  of 
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incidents  that  passed  this  criterion  are  shown  in  Table  1  broken  out  by  the  performance 
category  in  which  the  majority  of  judges  placed  the  incidents.  Altogether,  about  70%  of  the 
600  incidents  met  the  criterion. 

Examination  of  Table  1  reveals  that  over  half  of  the  421  incidents  that  passed  the  50% 
criterion  were  placed  in  four  performance  categories:  Enforcing  standards  of  behavior  and 
adherence  to  policies/regulations;  Maintaining  employee  morale  and  motivation;  Assuring 
unit  functioning;  and  Communicating  with  subordinates.  The  number  of  incidents  that  were 
placed  into  these  performance  categories  could  be  considered  an  index  of  the  relative 
criticality  of  the  categories  in  the  minds  of  the  workshop  participants,  as  frequency  and 
criticality  are  usually  highly  correlated  in  the  minds  of  judges'  (Sanchez  &  Levine,  1989). 
Using  that  index,  the  four  performance  categories  of  least  criticality  (that  combined  had  fewer 
than  20%  of  the  incidents  assigned  to  them)  were:  Showing  integrity  and  honesty;  Resolving 
employee  conflicts;  Completing  formal  employee  performance  appraisals;  and  Showing 
initiative,  extra  effort,  and  composure  under  stress. 

Designation  of  the  Performance  Levels  Reflected  by  Each  Incident 

As  mentioned  earlier,  the  retranslation  workshop  participants  indicated  the 
effectiveness  level  reflected  by  each  critical  incident  using  a  nine-point  scale,  where  "9"  was 
extremely  effective  and  "1"  was  extremely  ineffective.  The  standard  deviation  of  the 
effectiveness  rating  given  each  incident  by  the  judges  was  computed  for  the  incidents  that 
passed  the  50%  agreement  criterion.  If  the  standard  deviation  for  a  given  incident  was  equal 
to  or  less  than  1.333,  the  mean  effectiveness  of  the  incident  was  deemed  to  be  sufficiently 
stable  to  warrant  use  of  the  incident  in  constructing  the  scale  for  the  given  performance 
factor.  (With  16  judges,  a  standard  deviation  of  1.333  or  less  indicates  that  the  standard  error 
of  the  mean  effectiveness  rating  is  .333  or  less.)  Table  1  shows  the  number  of  incidents 
passing  the  50%  agreement  criterion  that  also  had  standard  deviations  of  1.333  or  less. 
Altogether,  over  75%  of  the  incidents  meeting  the  50%  agreement  criterion  met  the  1.333 
criterion. 

The  325  incidents  that  passed  both  the  50%  agreement  and  the  1.333  standard 
deviation  criteria  were  then  categorized  into  threo  nerformance  level  groups:  those  incidents 
that  had  mean  effectiveness  ratings  less  than  or  x  ual  to  3.667  were  placed  in  the  low 
performance  group  for  their  designated  perform:  va:e  factor;  those  with  mean  effectiveness 
ratings  between  3.668  and  6.333  were  placed  in  -he  medium  performance  group,  and  those 
incidents  with  mean  effectiveness  ratings  above  6.333  were  placed  in  the  high  performance 
group  for  their  designated  performance  factor.  Over  half  of  the  incidents  (57%)  were  placed 
in  the  high  performance  group,  while  about  40%  of  the  incidents  were  placed  in  the  low 
performance  group.  The  number  of  incidents  placed  in  the  medium  performance  group  was 
quite  small  --  less  than  3%  of  the  325  incidents. 


1  It  is  also  possible  that  the  number  of  critical  incidents  reflect  the  "observability"  of  the  behavior  subsumed 
within  the  category. 
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Table  1 


Number  of  Incidents  Placed  in  Each  Performance  Category  by  the  Retranslation 
Workshop  Participants 


Number  of 
Number  of  Incidents 

Incidents  Also  Passing 

Performance  Category  in  Which  Number  of  Passing  50%  Standard  Deviation 

Incident  Was  Most  Often  Placed  Incidents  Criterion  Criterion 


A 

Enforcing  Standards  of  Behavior  and 
Adherence  to  Policies/Regulations 

92 

78 

52 

B 

Maintaining  Safety,  Health,  and 

Physical  Well  Being 

32 

29 

26 

C 

Completing  Formal  Performance  Appraisals 

23 

20 

14 

D 

Resolving  Employee  Conflicts 

28 

19 

9 

E 

Providing  Personal  and  Career  Counseling 

43 

33 

21 

F 

Communicating  with  Subordinates 

65 

42 

38 

G 

Assuring  Employee  Technical  Competence 

50 

36 

26 

H 

Showing  Initiative,  Extra  effort,  and 
Composure  under  Stress 

32 

22 

22 

I 

Showing  Integrity  and  Honesty 

21 

15 

13 

J 

Maintaining  Employee  Moral  and  Motivation 

74 

52 

43 

K 

Assuring  Unit  Functioning 

72 

49 

38 

L 

Communicating,  Coordinating  with 
Supervisors  and  Other  Units 

30 

26 

23 

Total 

562* 

421 

325 

*  Total  does  not  equal  600  because  some  incidents  were  placed  equally  often  in  two  or  more  categories. 


15 


Detailed  Examination  of  the  Selected  Retranslated  Incidents 


The  specific  content  of  the  incidents  that  were  placed  in  the  high,  medium,  and  low  levels 
of  each  performance  category  were  examined  in  conjunction  with  the  behavioral  descriptions 
already  written  for  the  12  draft  performance  scales.  Content  discrepancies  and  shortfalls 
between  the  critical  incidents  and  the  behavioral  descriptors  used  in  the  draft  scales  were 
corrected  through  revisions  of  the  scale  behavioral  descriptors.  The  goal  was  to  ensure  that 
the  scale  performance  descriptors  encompassed  the  content  of  the  incidents  that  had  been 
reliably  placed  in  each  performance  category/level  combination.  As  very  few  incidents  had 
been  placed  in  the  medium  performance  level,  the  scale  descriptions  for  this  level  were  for 
the  most  part  statements  reflecting  performance  intermediate  between  the  high  and  low  scale 
descriptors. 

Evaluation  of  the  Intermediate  Supervisor  Performance  Factors 

Retranslation  workshop  participants  *  evaluations.  The  retranslation  workshop 
participants  evaluated  the  12  intermediate  performance  factors  on  the  following  factors: 
importance,  ease  of  rating,  applicability  across  different  types  of  jobs,  and  redundancy.  The 
four  performance  factors  selected  most  often  by  the  participants  for  inclusion  in  a  compre¬ 
hensive  overall  measure  of  first-level  supervisor  were  the  same  four  that  had  the  most  critical 
incidents,  namely,  Communicating  with  subordinates,  Enforcing  standards  of  behavior  and 
adherence  to  policies/  regulations.  Assuring  unit  functioning,  and  Maintaining  employee 
morale  and  motivation.  The  four  performance  factors  selected  least  often,  however,  were 
generally  not  the  ones  which  had  the  fewest  assigned  critical  incidents:  Providing  personal 
and  career  counseling;  Communicating,  coordinating  with  supervisors  and  other  units; 
Resolving  employee  conflicts;  and  Maintaining  employee  safety,  health,  and  physical  well 
being.  Two  of  the  four  least  selected  performance  factors  were  also  the  two  factors 
mentioned  most  often  as  being  difficult  to  rate  --  Resolving  employee  conflicts  and  Providing 
personal  and  career  counseling. 

The  factor,  Providing  personal  and  career  counseling,  was  also  judged  by  many 
participants  to  be  measuring  the  same  factor  as  Completing  formal  employee  performance 
appraisals.  Two  pairs  of  performance  factors  that  were  similarly  judged  by  many  participants 
to  overlap  were  Communicating,  coordinating  with  supervisors  and  other 
units/Communicating  with  subordinates  and  Assuring  unit  functioning/Assuring  technical 
competence  of  subordinates. 

None  of  the  12  factors  were  mentioned  as  being  particularly  inappropriate  for 
measuring  the  effectiveness  of  first-level  supervisor  performance  in  certain  types  of  jobs. 

Draft  scale  tryout  results.  The  tryout  of  the  draft  scales  was  somewhat  atypical. 

The  workshop  participants  were  free  to  choose  their  ratees.  The  ratees  themselves  were 
unidentified  except  by  their  initials.  And  the  number  of  raters  (12)  and  ratees  (36)  was  small. 
The  reader  is  cautioned,  therefore,  not  to  place  much  reliance  on  the  results  obtained,  though 
they  were  quite  encouraging. 
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The  range  of  the  ratings  of  the  36  supervisors  covered  the  entire  7-point  scale  for  most 
of  the  12  scales;  for  the  remainder  of  the  scales  the  range  of  the  ratings  covered  6  points  on 
the  scale.  The  median  standard  deviation  of  the  ratings  on  the  12  scales  was  about  1.6, 
another  indication  that  the  raters  spread  their  ratings  across  the  7-point  scales.  The  mean 
ratings  across  all  scales  was  about  4.7,  indicating  that  on  the  average,  the  36  supervisors  were 
rated  somewhat  above  the  midpoint  of  the  scales,  but  not  too  highly. 

The  intercorrelations  among  the  12  scales  ranged  from  .29  to  .87.  A  principal 
component  factor  analysis  of  the  intercorrelation  matrix  indicated  that  one  factor  may  underlie 
the  relationships  among  the  scales.2  The  tryout  results  indicate  that  a  composite  overall  score 
derived  by  summing  the  12  ratings  for  a  given  supervisor  would  probably  capture  the 
performance  of  the  supervisors  being  rated  without  the  need  for  subscores. 

The  tryout  results  also  indicated  that  an  overall  performance  score  derived  by 
summing  the  ratings  across  12  scales  would  probably  be  highly  reliable.  The  Alpha 
reliability  for  the  total  rating  score3  obtained  with  the  36  cases  was  .96.  Because  of  the  high 
degree  of  relationship  among  the  12  scales,  one  or  two  of  the  scales  could  probably  be 
dropped  without  undue  loss  in  reliability. 

The  Scales  Selected  for  Use  in  the  Trial  Administration 


Although  the  scale  tryout  data  suggested  that  one  or  more  scales  could  be  dropped 
with  little  loss  in  the  comprehensiveness  and  reliability  of  a  total  performance  score  based  on 
the  scales,  it  was  decided  to  pilot  test  all  the  scales.  It  was  felt  that  until  actual  ratings  of 
first-level  supervisors  were  obtained  from  their  own  supervisors,  issues  concerning  the 
difficulty  of  making  the  ratings,  the  factor  structure  and  reliability  of  the  ratings,  and  scale 
redundancy  could  not  be  resolved  definitively.  The  scales  selected  for  use  in  the  trial 
administration  are  given  in  Appendix  C. 

oince  reviewers  of  the  12  scales  suggested  that  the  scale,  Showing  initiative,  extra 
effort,  and  composure  under  stress,  should  be  subdivided,  13  scales  are  given  in  Appendix  C. 
The  reviewers  felt  that  composure  under  stress  was  essentially  a  different  aspect  of 
performance  than  showing  initiative  and  extra  effort.  The  original  scale  was  therefore 
subdivided  into  two  scales.  Figure  3  shows  the  single  scale  and  the  two  scales  derived  from 
it.  The  critical  incidents  placed  by  the  retranslation  workshop  participants  into  the  original 
performance  category  were  subdivided  and  used  in  the  same  manner  described  earlier  as  a 
guide  in  writing  the  performance  level  scale  descriptors. 

Of  interest  is  the  degree  of  correspondence  between  the  dimensions  of  supervisory 
performance  captured  by  the  13  scales  and  the  25  preliminary  performance  factors  identified 
as  a  result  of  the  earlier  examination  of  the  responsibilities  and  tasks  of  first-level  civilian 


1  Only  the  first  factor  extracted  had  an  eigenvalue  equal  to  or  greater  than  1.0. 

3  The  reader  is  reminded  that  such  an  overall  score  would  also  probably  contain  some  invalid  variance,  e.g.,  rater 
"halo"  error. 


17 


H.  SHOWING  INITIATIVE,  EXTRA  EFFORT,  AND  COPOSURE  UNDER  STRESS 
How  effective  is  this  supervisor  in  showing  initiative,  extra  effort,  and  composure  under  stress? 
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supervisors  working  for  the  federal  government  (see  Appendix  A).  The  content  of  all  13 
scales  was  reflected  in  part  or  whole  in  the  list  of  25  performance  factors.  However,  not  all 
the  preliminary  performance  factors  were  reflected  in  the  13  scales.  Specifically,  four  of  the 
performance  factors  were  not  represented  in  the  scales: 

Maintaining  security  of  sensitive  or  classified  information; 

Recommending  the  selection,  promotion,  and  reassignment  of  subordinates; 
Displaying  technical  knowledge  and  skill;  and 
Developing  own  job  and  supervisory  skills. 

The  critical  incidents  produced  by  the  workshop  participants  did  not  capture  these  aspects  of 
first-level  civilian  supervisor  performance  sufficiently  to  warrant  their  inclusion  in  the  scales. 


Administration  of  the  Trial  Scales 


The  trial  performance  evaluation  scales  were  administered  to  the  first-  and  second- 
level  supervisors  of  participants  in  workshops  in  which  the  scoreable  in-basket  exercises  were 
pilot  tested.  All  the  workshop  participants  were  first-level  supervisors.  A  total  of  27 
supervisors  participated  in  the  GS-9  and  below  workshops  and  26  supervisors  participated  in 
the  GS  10-13  workshops.  Table  2  presents  descriptive  data  on  the  workshop  participants. 

Complete  sets  of  ratings  were  not  obtained  for  all  workshop  participants.  Only  first- 
level  supervisor  ratings  were  obtained  for  13  participants  and  only  second-level  supervisor 
ratings  were  obtained  for  five  participants.  Of  the  35  participants  for  whom  both  first-  and 
second-level  supervisor  ratings  were  obtained,  33  had  complete  sets  of  13  ratings  by  each 
type  of  supervisor.  (Two  first-level  supervisors  failed  to  provide  ratings  on  all  13  scales  for 
their  subordinates). 

Analyses  were  conducted  to  determine  whether  the  rating  scales  taken  as  a  set  were 
sufficiently  promising  from  a  measurement  viewpoint  to  warrant  use  in  the  validation  of  the 
scoreable  in-basket  exercises  and  the  biographical  data  and  temperament  instruments.  The 
analysis  of  the  rating  data  had  four  major  purposes: 

1)  To  determine  whether  the  13  rating  scales  were  essentially  measuring  one 
overall  performance  factor  or  two  or  more  separate  performance  factors; 

2)  To  determine  the  reliability  of  composite  scores  derived  from  combining  or 
averaging  ratings  on  several  or  all  of  the  13  scales; 

3)  To  determine  the  degree  of  similarity  between  the  ratings  made  by  first-level 
supervisors  and  second-level  supervisors.  (It  is  anticipated  that  there  will  be 
sizeable  number  of  participants  in  the  validation  study  for  whom  rating  data 
from  only  one  type  of  supervisor  will  be  available);  and 

4)  To  examine  differences  between  the  ratings  received  by  participants  in  Grades 
9  and  below  and  Grades  10-12. 
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Table  2.  Scoreable  In-Basket  Workshop  Participants1 


GS9  and  Below  GS10-13 

Workshop  Workshop 

Participants  Participants 


Number 

Percent 

Number 

Percent 

Location 

Ft.  Sam  Houston,  TX 

7 

25.9 

8 

30.8 

Ft.  Devon,  MA 

9 

33.3 

10 

38.5 

LMVD,  MS 

11 

40.7 

7 

26.9 

Data  Missing 

Total 

27 

* 

1 

26 

3.8 

Gender 

Male 

15 

55.5 

14 

53.9 

Female 

9 

33.3 

11 

42.3 

Missing 

3 

11.1 

1 

3.8 

Race/National  Origin 

White 

14 

51.8 

22 

84.6 

Black 

5 

18.5 

1 

3.8 

Hispanic 

6 

22.2 

1 

3.8 

Asian 

0 

- 

- 

- 

Other 

1 

3.7 

- 

• 

Data  Missing 

1 

3.7 

2 

7.7 

Pay  Plan 

GS 

17 

62.9 

15 

57.7 

GM 

- 

- 

6 

23.1 

WS 

4 

14.8 

2 

7.7 

Data  Missing 

6 

22.2 

3 

11.5 

Length  of  Time  as  Supervisor 

5  years  or  less 

13 

48.1 

12 

46.2 

6-10  years 

5 

18.5 

5 

19.2 

11-15  years 

5 

18.5 

4 

15.4 

16-20  years 

2 

7.4 

2 

7.7 

21-25  years 

1 

3.7 

2 

7.7 

Data  Missing 

1 

3.7 

1 

3.8 

No.  of  Subordinates  Supervised 

1-5 

9 

33.3 

6 

23.1 

6-10 

8 

29.6 

12 

46.2 

11-15 

4 

14.8 

4 

15.4 

More  than  15 

3 

11.1 

3 

11.5 

Data  Missing 

3 

11.1 

1 

3.8 

1  This  table  was  extracted  from  a  more  complete  table  of  workshop  participant  characteristics  given  in  Selecting 
First-Level  Supervisors:  The  Development  of  Two  Scoreable  In-Basket  Exercises  (HumRRO  Final  Report  FR- 
PRD-93-03),  by  H.R.  Felber,  S.M.H.  Sandlund,  B.A.  Dugan,  and  C.K.  Rigby,  March  1993. 
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Separate  principal  components  factor  analyses  were  conducted  on  the  13  supervisory 
scales  using  the  ratings  obtained  respectively  from  the  first-level  supervisors  of  ratees,  the 
second-level  supervisors  of  the  ratees,  and  the  average  of  the  first-  and  second-level  ratings  of 
the  ratees.  Similar  to  the  results  obtained  in  the  earlier  tryout  of  the  scales,  in  each  of  the 
three  factor  analyses  only  one  factor  met  the  criterion  of  having  an  eigenvalue  equal  to  or 
greater  than  1.0.  Table  3  shows  the  factor  loadings  obtained  for  the  13  scales  in  the  three 
analyses.  These  loadings  for  the  most  part  were  quite  high,  averaging  above  .80  across  the 
three  analyses. 

As  might  be  expected  from  the  undimensional  factor  analytic  results  and  the  tryout 
reliability  results  (see  page  24),  composites  obtained  through  averaging  the  13  separate  ratings 
received  by  the  participants  had  high  reliabilities.  The  Alpha  reliability  of  the  composite 
obtained  by  averaging  the  13  ratings  made  by  the  first- level  supervisors  of  the  participants 
was  .95.  The  Alpha  reliability  for  the  comparable  composite  obtained  from  the  second-level 
supervisor  ratings  was  also  .95,  while  the  Alpha  reliability  of  the  composite  obtained  from 
first  averaging  the  first-  and  second-level  supervisory  ratings  on  each  scale  was  .97.  Further 
analyses  indicated  that  dropping  any  one  of  the  13  scales  from  the  composites  would  likely 
lower  the  reliability  of  the  composites,  though  only  slightly. 

The  first-level  supervisor  composite  rating  had  a  high  correlation  (.86)  with  the 
second-level  supervisor  composite  rating  across  the  33  participants  from  which  both  compos¬ 
ites  were  available.4  The  correlation3  of  the  first-level  supervisor  composite  with  the 
composite  obtained  from  averaging  the  first-  and  second-level  supervisor  ratings  was  .97. 

The  comparable  correlation3  of  the  second-level  supervisor  composite  with  the  average  rating 
composite  was  .96.  These  results,  coupled  with  the  factor  analytic  and  reliability  results  cited 
above,  were  considered  to  be  quite  encouraging.  The  results  indicated  that  reliable,  inter¬ 
changeable  composites  could  be  formed  from  either  the  first-  or  the  second-level  supervisory 
ratings  in  the  absence  of  one  or  the  other  set  of  ratings.  Furthermore,  averaging  the  first-  and 
second-level  ratings  (when  both  sets  of  ratings  were  available),  could  produce  a  composite 
with  very  high  reliability  and  high  correlations  with  composites  based  on  the  ratings  made  by 
only  one  type  of  supervisor.  (Not  infrequently  in  large  scale  data  collections,  ratings  are 
unavailable  from  both  the  first-  and  second-level  supervisors  of  some  of  the  ratees.) 

A  repeated  measure  analysis  of  variance  was  conducted  to  determine  whether  the 
mean  differences  in  ratings  received  by  participants  in  Grades  9  and  below  were  significantly 
different  from  those  received  by  participants  in  Grades  10-13.  The  analysis  also  determined 
whether  the  ratings  given  by  the  first-level  supervisors  were  on  the  average  significantly 
different  from  those  given  by  the  second-level  supervisor  and  whether  the  participants  on  the 
average  received  higher  or  lower  ratings  on  some  of  the  13  scales  than  on  others.  Interaction 
effects  were  also  tested  in  the  analysis  of  variance. 


4  The  correlation  of  .86  between  the  first-  and  second-level  supervisor  composite  ratings  is  quite  high,  and  may 
in  pert  be  attributable  to  sampling  error  (the  sample  size  was  only  33  ratees). 

1  The  reader  should  bear  in  mind  that  these  correlations  represent  part/whole  relationships. 
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Table  3.  Factor  Analyses  of  First-Level,  Second-Level,  and  Average  Supervisor  Ratings  of  the  Trial  Sample  Participants 


00 
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5 


22 


[.  Performing  Well  Under  Pressure  and  Stress 


The  results  of  the  analysis  of  variance  were  encouraging  (see  Table  4).  The  mean 
ratings  (across  the  13  scales)  assigned  by  the  two  types  of  raters  (first-  and  second- level 
supervisors)  to  their  subordinate  supervisors  were  not  significantly  different.  That  is,  on  the 
average  the  first-level  supervisors  of  the  workshop  participants  did  not  assign  significantly 
higher  or  lower  ratings  than  the  second-level  supervisors.  Nor  was  there  any  evidence  that 
the  two  types  of  raters  tended  to  rate  the  lower  and  higher  grade  supervisors  in  the  sample 
differentially  (the  rater  type  interactions  with  the  ratee  grade  level  and  scale  factors  were  not 
significant).  These  results,  if  confirmed  in  future  analyses  with  larger  samples  of  ratees  and 
raters,  further  point  to  the  interchangeability  of  the  first-  and  second-level  supervisor  ratings. 

It  is  interesting  to  note  that  the  between  subjects  effect  in  the  analysis  of  variance  was 
not  significant.  That  is,  the  GS  10  to  13  supervisors  in  the  workshop  sample  did  not  receive 
significantly  higher  or  lower  ratings  on  the  average  than  the  GS  9  and  below  supervisors. 
There  was,  however,  evidence  that  the  lower  grade  supervisors  received  higher  ratings  on 
some  of  the  13  scales  and  lower  ratings  on  other  scales  than  the  higher  grade  supervisors  did 
(the  Scale  X  Grade  interaction  term  was  significantly  different  from  zero).  Table  5  shows  the 
average  ratings  received  by  the  participants  broken  out  by  participant  grade  level  and  scale. 

The  most  statistically  significant  result  obtained  in  the  repeated  measure  analysis  of 
variance  concerned  the  differences  in  the  mean  ratings  assigned  the  participants  on  the  various 
scales  (see  Table  4).  The  scale  mean  differences  were  significant  at  the  .0001  level.  The 
participants  as  a  whole  received  the  lowest  average  ratings  on  the  scale.  Showing  initiative 
and  extra  effort  (see  Table  5).  The  participants  received  their  highest  ratings  on  the  average 
on  the  scale,  Assuring  technical  competence  of  subordinates.  These  rating  differentials,  if 
confirmed  in  analyses  with  larger  samples,  could  point  to  areas  in  which  Army  civilian 
personnel  managers  might  focus  staff  performance  development  activities. 

The  range  of  the  ratings  given  by  the  supervisors  of  the  workshop  participants 
covered  the  entire  7-point  scale  for  most  of  the  13  scales.  The  first-level  supervisors  used  the 
entire  range  in  12  of  the  13  scales.  The  second-level  supervisors  used  the  entire  range  in  9  of 
the  13  scales.  The  median  standard  deviation  of  the  ratings  on  the  13  scales  was  about  1.3 
for  both  groups  of  raters.  The  standard  deviation  of  the  average  ratings  (across  all  13  scales 
and  both  types  of  raters)  was  1.05.  The  average  ratings  ranged  from  1.42  to  6.65.  Both  the 
ranges  and  standard  deviations  obtained  indicate  that  the  ratings  of  the  workshop  participants 
were  spread  across  the  separate  and  combined  scales. 

The  overall  (across  scales  and  raters)  mean  rating  of  the  workshop  participants  was 
5.46.  This  somewhat  high  value  indicates  that  the  raters  tended  to  assign  fairly  high  ratings 
to  the  participants.  As  the  participants  were  selected  by  their  organizations  on  a  non-random 
basis,  the  high  ratings  may  reflect  their  actual  performance  levels.  Care  should  be  exercised 
in  the  administration  of  the  scales  to  emphasize  to  raters  the  importance  of  making  accurate 
ratings  of  their  subordinates  ^nd  avoiding  halo  and  other  kinds  of  errors  that  raters  typically 
make.  The  use  of  behavioral  anchors  in  the  scales  hopefully  will  encourage  more  objective 
performance  judgments. 
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Table  4.  Results  of  Repeated  Measure  Analysis  of  Variance  (n  =  33) 


Source  of  Variation 

Degree  of 
Freedom 

Sum  of 
Squares 

Mean 

Square 

F  Ratio 

Probability 

Between  Subject  Effect 

Grade  Level  (G) 

1 

7.83 

7.83 

.27 

n.s. 

Error  (Grade) 

31 

903.09 

29.13 

Within  Subject  Effects 

Scale  (S) 

12 

55.02 

4.58 

6.35 

.0001 

S  x  G 

12 

16.99 

1.42 

1.96 

.0266 

Error  (Scale) 

372 

268.45 

.72 

Rater  Type  (R) 

1 

2.06 

2.06 

.92 

n.s. 

Rx  G 

1 

2.61 

2.61 

1.16 

n.s. 

Error  (Rater  Type) 

31 

69.64 

2.25 

S  x  R 

12 

8.49 

.71 

1.14 

n.s. 

S  x  R  x  G 

12 

8.73 

.73 

1.17 

n.s. 

Error  (S  x  R) 

372 

230.48 

.62 

Summary  and  Conclusions 

In  summary,  the  psychometric  analyses  of  the  rating  scale  data  suggested  that  the 
scales  would  be  sufficiently  sound  to  use  in  the  validation  of  the  scoreable  in-basket  exercises 
and  the  biographical  data  and  temperament  instruments.  The  reliability  of  performance 
evaluation  scores  obtained  from  averaging  the  separate  ratings  on  the  13  scales  is  likely  to  be 
quite  high.  The  ratings  given  by  the  first-  or  second-level  supervisors  were  sufficiently  alike 
to  warrant  expectations  that  the  two  sets  of  ratings  could  be  either  combined  to  create  an  even 
more  reliable  composite,  or  used  interchangeably  in  case  rating  data  are  collected  from  only 
one  supervisor  of  a  given  participant. 

Although  the  analysis  indicated  that  the  scales  may  be  measuring  one  overall 
performance  dimension  or  factor,  and  that  dropping  one  or  two  scales  would  have  minimal 
effect  on  die  psychometric  quality  of  the  scales,  it  was  decided  to  use  all  13  scales  in  the 
validation.  Plans  for  the  validation  called  for  the  administration  of  a  set  of  six  independently 
developed  scales  along  with  the  behaviorally  anchored  scales  described  in  this  report. 
Obtaining  ratings  on  all  13  scales  would  allow  the  factorial  structure  of  both  sets  of  scales  to 
be  more  fully  explored.  In  addition,  retaining  all  the  scales  preserved  their  comprehensive¬ 
ness. 
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Table  5.  Average1  Ratings  Received  by  Participants 
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Averages  include  both  first-  and  second-level  supervisor  ratings. 


Ultimately,  the  most  important  issue  in  regard  to  the  scales  is  whether  they  adequately 
reflect  the  performance  requirements  of  Army  first-level  civilian  superp 

mber  of  Army  civilian  supervisors  in  both  the  initial  generation  of  the  critical  incidents  and 
their  later  placement  into  performance  categories  further  served  to  assure  the  validity  of  the 
scales.  The  review  and  evaluations  of  the  scales  by  Army  civilian  personnel  experts  also 
helped  establish  the  validity  of  the  scales. 

In  conclusion,  considering  both  how  the  performance  evaluation  scales  were  devel¬ 
oped  and  the  available  data  concerning  their  psychometric  characteristics,  the  authors 
recommend  their  use  as  criteria  in  the  validation  of  selection  instruments  for  Army  civilian 
first-level  supervisors.  Ratings  on  the  scales  could  also  be  used  as  baseline  measures  for  the 
evaluation  of  civilian  supervisory  performance  over  time.  For  example,  ratings  of  supervisors 
could  be  obtained  both  before  and  after  the  application  of  a  personnel  initiative  to  determine 
whether  the  change  had  the  desired  effect  of  raising  performance. 
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APPENDIX  A 

MATERIALS  USED  IN  CRITICAL  INCIDENT  WORKSHOPS 


A-l 


Supervisor  Job  Performance  Workshop 

Time* 

Topic 

0800  -  0820 

Overview  of  the  project.  Description  of 
workshop  activities. 

0820  -  0840 

Training  on  writing  critical  incidents 

0840  -  0950 

Generating  critical  incidents 

0950  -  1000 

Break 

1000  -  1110 

Generating  more  critical  incidents 

1110  -  1140 

Evaluate  preliminary  set  of  performance 
measurement  factors 

1140  -  1200 

Discussion  of  measurement  factors  and  problems 

^Time  periods  depended  upon  workshop  starting  time. 
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ARMY  CIVILIAN  PERSONNEL  RESEARCH 
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ORIENTATION  TO  CIVPERS  PROJECT 


Most  managers  would  agree  that  the  civilian  first-line  supervisor 
plays  a  critical  role  in  the  Army  and  that  the  selection  of  the  best  first- 
line  supervisor  candidates  is  an  essential  component  of  Army  readiness. 
However,  officials  responsible  for  selection  have  difficulty  estimating  the 
supervisory  potential  of  candidates  because  few  applicants  have  ever  worked 
as  supervisors.  As  a  result,  in  many  cases  the  overriding  factor  in  a 
candidate's  selection  is  technical  competence  rather  than  supervisory 
potential.  A  research  effort  has  been  recently  initiated  by  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  to  evaluate  the 
supervisory  aspects  of  the  job  of  civilian  first-line  supervisors  in  the 
Army  and  to  develop  a  set  of  supervisory  selection  measures  which  can  more 
accurately  assess  supervisory  potential. 

To  help  accomplish  this  research,  ARI  has  retained  a  team  of 
contractors  led  by  the  Human  Resources  Research  Organization  (HumRRO).  The 
research  team  is  composed  of  the  following  organizations:  HumRRO,  Caliber 
Associates,  Systems  Research  and  Applications  Corporation  (SRA),  and  Booz- 
Allen  Hamilton.  Our  research  team  is  known  for  developing  products  that 
are  immediately  useful  to  sponsors. 

Today,  we  will  first  review  what  types  of  first-line  supervisory 
selection  instruments  are  going  to  be  developed  for  Civilian  personnel, 
then,  we  would  like  your  help  in  providing  us  with  crucial  information  that 
will  aid  us  in  the  development  of  these  Civilian  First- line  supervisory 
selection  instruments. 
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CANDIDATE  SELECTION 
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ANSWER:  Valid  2  Part  Selection  Procedure  for 

First  Line  Supervisors 

•  Structured  Interview  Guide 

•  Scoreable  In-Basket  Exercise 


CANDIDATE  SELECTION 


Candidate  selection  work  addresses  the  need  to  select  first- line 
supervisors  with  strong  supervisory  potential.  Recognizing  that  over 
26,000  first-line  supervisors  currently  serve  in  the  DA  civilian 
workforce  and  that  over  3000  first- line  supervisor  promotions  are  made 
annually,  we  made  development  of  valid  selection  procedures  a  top 
priority. 

Our  research  and  development  efforts  indicate  that  the  need  for 
valid  selection  procedures  will  be  best  met  by  at  least  two  procedures: 

•  A  Structured  Interview  Guide  -  designed  to  specify  crucial 
topics  to  address  and  assess  in  the  personal  interview  setting. 

•  A  Scoreable  In-basket  Exercise  -  a  judgment  and  decision-making 
exercise  with  scoring  criteria  to  reduce  administration  time. 
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V 


I 


WHY  THESE  TWO  SELECTION  PROCEDURES? 

These  selection  procedures  were  chosen  for  two  particular  principles. 

•  Validity  -  The  job  requirements  of  the  first-line  supervisors 
(the  knowledges,  skills,  abilities  and  other  characteristics) 
were  thoroughly  documented  and  matched  to  the  procedures  best 
suited  to  measure  each  job  requirement. 

•  Usability  -  Selection  procedures  were  chosen  with  regard  to 
their  ease  of  use,  reduced  administration  time,  and  potential 
for  integration  into  existing  Army  civilian  selection  procedures 
including  computerized  systems. 
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WORKSHOP  OVERVIEW 


In  order  to  determine  whether  scores  on  the  selection  instruments  are 
related  to  on-the-job  performance,  it  is  vital  to  determine  whether  first- 
line  supervisors  who  do  well  on  the  selection  instruments  also,  in  fact,  do 
well  on  the  job.  Conversely,  do  low  scorers  on  the  selection  tests  perform 
less  effectively  in  supervisory  jobs?  To  answer  these  questions,  we  need  to 
obtain  an  accurate  picture  of  the  job  performance  of  individual  first- line 
supervisors. 

Unfortunately,  the  standard  rating  forms  used  to  evaluate  Army 
supervisors  are  not  likely  to  be  of  much  help  here.  For  various  reasons, 
most  of  the  ratings  tend  to  be  all  bunched  up  at  the  successful  end  of  the 
scale.  This,  of  ccjrse,  makes  it  impossible  to  tell  who's  actually 
performing  effectively  and  who  is  performing  less  effectively.  A  special 
evaluation  of  individual  supervisor  performance  is  needed,  an  evaluation 
that  doesn't  go  into  any  official  personnel  record,  but  is  used  for  research 
purposes  only.  This  actual  performance  rating,  in  turn,  requires  a  special 
rating  form  developed  to  help  raters  provide  an  accurate  picture  of  an  Army 
civilian  supervisor's  performance  and  effectiveness.  This  is  where  you  and 
these  workshops  come  in. 

We  are  going  to  work  together  to  develop  a  state-of-the-art  rating  form. 
This  form  should  help  raters  (for  example,  second-line  supervisors  or 
fellow  first-line  supervisor)  make  accurate  judgments  of  the  actual 
performance  effectiveness  of  supervisors  they  either  supervise  or  work  with. 
Together,  we  will  design  what  are  referred  to  as  behavior-based  rating 
scales,  a  rating  form  that  offers  an  opportunity  for  relatively  objective 
assessments  of  performance. 

Before  introducing  the  behavior-based  rating  scale  concept,  here  are 
some  other  kinds  of  rating  forms. 
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1.  An  example  of  trait  ratings: 


Leadership 


* 


Poor 


Good 


H 

Low 


Aggressiveness 

—7^— 


High 


Self-Confidence 


H 

Low 


— I 

High 
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2.  An  example  of  numerically  anchored  ratings: 


a. 


Quality  of  Work 
2  3  Q)  5 


b.  Quality  of  Work:  Judge  the  amount  of  scrap,  consider  general 

care  and  accuracy  of  work. 


poor,  1-6;  average,  7-18;  good,  19-25 
£J2- 


c. 


Quality 


1"  2  3  TT 

-mr-T nr 

'IT  T2  13"T4'T5 

IS  17 I'd  15  26 

ITTflTTT 

1  1  l  1 

iiii 

IIII 

IXI  1  1 

1  1  1 

POOR 

BELOW 

AVERAGE 

AVERAGE 

avISXIe 

EXCELLENT 
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3.  Examples  of  behavior-based  rating  scales 


CONSIDERATION  FOR  SUBORDINATES 


How  effective  is  the  supervisor  in  showing  consideration  toward  and 

providing  help  to  subordinates? 


Ignores  subordinates' 
feelings  and  refuses  or 
otherwise  fails  to  help 
with  work-related  or 
personal  problems;  often 
neglects  to  pass  on 
Important  Information  to 
subordinates. 


1  2 


In  most  situations, 
helps  subordinates  with 
their  work-related/ 
personal  problems; 
usually  informs  unit 
members  about  what  they 
need  to  know. 


3  4  5 


Takes  extra  steps  to 
help  subordinates  with 
work-related  and 
personal  problems; 
always  informs  unit 
members  concerning 
information  they  should 
know  about. 


6  7 


_ SELF-DEVELOPMENT _ 

How  effective  is  the  supervisor  in  developing  own  job 
and  leadership  skills? 


Makes  little  or  no 
effort  to  improve 
technical /leadership 
skills; 

refuses  or  otherwise 
fails  to  participate 
in  relevant  training 
courses  or  career 
growth  opportunities. 


1  2 


Studies,  practices,  or 
participates  in 
course/training  when 
required  to  do  so  or 
when  encouraged  to  work 
on  problem  areas. 


3  4  5 


Takes  courses,  studies, 
and  works  hard  during 
non-work  hours  to 
improve  technical  and 
leadership  skills; 
actively  seeks 
additional  job  duties/ 
responsibilities  to 
improve  job  skills  and 
increase  chances 
of  promotion. 


6  7 


A-12 


Advantages  of  Behavior-Based  Rating  Scales 


1.  The  rating  task  with  these  scales  emphasizes  objective  observation 
rather  than  subjective  evaluation. 

2.  Scales  constructed  to  reflect  performance  requirements  regarded  as 
important  by  those  knowledgeable  about  the  job. 

3.  Scales  define  the  relevant  and  important  performance  requirements  in 
concrete  terms. 


4.  Job  experts  agree  on  the  effectiveness  levels  of  scaled  job  behaviors 
used  as  performance  effectiveness  "anchors." 

5.  In  sum,  raters  can  compare  the  observed  performance  of  a  supervisor  to 
behavioral  benchmarks  or  standards  of  effectiveness,  resulting  in  more 
objective  performance  judgments. 
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HOW  TO  WRITE  FIRST  LINE  SUPERVISOR  EFFECTIVENESS  INCIDENTS 


To  write  a  performance  example  or  Incident,  try  to  remember  what  the  first 
line  supervisor  actually  did  or  failed  to  do  that  made  him  or  her  effective 
or  Ineffective  In  a  situation.  These  can  be  examples  of  extremely 
effective,  ineffective,  or  even  average  performance.  The  important  thing  is 
that  the  incident  is  described  specifically  as  it  happened . 

When  writing  an  Incident,  describe  only  what  you  saw  or  what  the  person  did, 
not  what  you  inferred  from  the  action.  For  example,  in  writing  an  incident, 
rather  than  writing  that  the  supervisor  "displayed  conscientiousness,"  you 
should  describe  what  this  supervisor  did  to  make  you  believe  he  or  she  was 
conscientious.  As  examples,  the  supervisor  "worked  all  night  to  accomplish 
a  job,"  or  "came  In  very  early  to  set  up  work  assignments."  Both  of  these 
behaviors  or  actions  might  be  described  as  displays  of  conscientiousness; 
they  are  things  a  supervisor  did  to  make  the  writer  believe  he  or  she  was 
conscientious.  Thus,  we  are  asking  you  to  describe  specific  behaviors  or 
actions,  not  traits  or  personal  characteristics. 

The  features  of  a  good  Incident  are: 

1.  It  concerns  the  actions  of  an  individual  first  line  supervisor  in 
his/her  job. 

2.  It  tells  what  the  supervisor  did  (or  did  not  do)  that  made  you  feel  he 
or  she  was  effective  or  ineffective. 

3.  It  describes  clearly  the  background  of  the  incident. 

4.  It  states  consequences  of  what  the  supervisor  did. 

5.  It  is  concise  in  that  it  is  short,  to  the  point,  and  does  not  go  to 
great  lengths  specifying  unimportant  details  of  the  background,  the 
activity  itself,  or  the  consequences  of  what  the  supervisor  did. 


On  the 
"up  to 


following  pages  are  some  hypothetical  examples  we  will  use  to  get  you 
speed"  to  write  behavioral  examples  of  incidents. 


A-14 


SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1. 


What  were  the  circumstances  leading  up  to  the  Incident? 

A  btUe^ing  has  to  be.  given. 


Too  little 
Information 


2. 


What  did  the  Individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 


The  supe/cvisotc  gave  it  his  all  and  did  a  good  job. 


Does  not 
describe  how 
the  supervisor 
“gave  It  his 
all",  or  wiiat 
he  did  that  was 
a  "good  job" 


3.  What  was  the  outcome  of  this  Incident? 

The  wotik  o£  the  section  was  increased. 


It's  not 
clear  what 
was  the 
result  of  the 
supervisor's 
action 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 

PeA^o tuning  well  undeti  sttieis. 


5.  Circle  the  number  below  that  best  reflects  the  correct  effectiveness  level  for 
this  example. 

1  2  3  4  5  6  7  (§)  9 


extremely 

ineffective 

about 

ineffective 

average 

effective  extremely 

effective 
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SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  were  the  circumstances  leading  up  to  the  incident? 

The  Impontance  oh  the  section’s  wonk  i oas  being  questioned  by  higher 
headquasitens .  The  supenvlson  was  given  pnactlcally  no  wanning,  but  was 
suddenly  called  In  to  explain  what  the  section  did. 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 

In  the  house's  time  available,  the  supenvlson  gathered  all  necessany  | facts 

and  hl9wn.es  and  presented  them  cleanly  and  compelllngly  to  the  ohhlc^als  pnesent. 


3.  What  was  the  outcome  of  this  incident? 

The  section  was  given  even  mone  nesponslblllty  and  eventually  two  mone  slots 
on  positions. 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 

Penhonmlng  well  unden  stness. 


Circle  the  number  below  that  best 
this  example. 

reflects 

the  correct  effectiveness  level  for 

12  3  4 

5 

6  7 

0 

9 

extremely  ineffective 

ineffective 

about 

average 

effective 

extremely 

effective 
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SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1. 


What  were  the  circumstances  leading  up  to  the  Incident? 
The  "No  Smoking "  iign6  voexe.  being  ignon&d. 


Insufficient 
Information  to 
evaluate  the 
superv i sor ' s 
behavior 


2. 


What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 


Vid  not  entfo/tce  the.  no  smoking  note. 


Doesn't  clearly 
describe  what 
the  supervisor 
did 


3.  What  was  the  outcome  of  this  incident? 
A  iviz  itasited. 


Doesn't  give 

enough 

information 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 
Maintaining  employze.  Aa^zty. 


Circle  the  number  below  that  best  reflects 
this  example. 

the  correct  effectiveness  level 

1  2 

Q  4 

5 

6  7 

8  9 

extremely 

ineffective 

ineffective 

about 

average 

effective 

extremely 

effective 
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SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  were  the  circumstances  leading  up  to  the  Incident? 

The  thop  contained  a  lot  oh  ^taomablz  matexialA  and  had  "Wo  Smoking"  <s ignt> 
ported,  but  Aome  oh  the  woxkeu  smoked  anyway. 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 

He  did  not  enhoxce  the.  tahety  xules.  He  dimply  ignoxed  the  cigaxette  smoking. 


3.  What  was  the  outcome  of  this  incident? 

A  lighted  cigaxette  xolled  ohh  a  thelh  into  a  can  oh  hlwmmable  liquid  and 
qtaxted  a  | \ixe.  Luckily,  it  wat>  quickly  put  out. 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 

Maintaining  employee  Aahety. 


Circle  the  number  below  that  best  reflects 
this  example. 

the  correct  effectiveness  level 

CNJ 

pH 

5 

6  7  8 

9 

extremely  ineffective 

ineffective 

about 

average 

effective 

extremely 

effective 
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SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  were  the  circumstances  leading  up  to  the  incident? 

V.E.  was  o  new  employee  who  thought  he  had  a  way  with 
women.  So  he  slanted  making  suggestive  nema/cks  to  the 
section  secnetany,  who  was  pnetty,  but  mounted,  and 
wouldn't  give  him  the  time  o£  day. 


Includes 

irrelevant 

information 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 

The  supenvison.  showed  good  judgment  In  handling  the 
situation. 


3.  What  was  the  outcome  of  this  incident? 


V.E.  stopped  making  suggestive  nemanks  to  the 
secnetany.  The  secnetany  neven  did  frile  a  complaint. 


No  names 
please 


Labels  the 
behavior.  Does 
not  indicate 
what  the  super¬ 
visor  actually 
d,d 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 
Resolving  problems  among  employees. 


5.  Circle  the  number  below  that  best  reflects  the  correct  effectiveness  level  for 
this  example. 


12  3  4 

5 

6  Q  8 

9 

extremely  ineffective 

about 

effective 

extremely 

ineffective 

average 
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effective 

SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  were  the  circumstances  leading  up  to  the  Incident? 

A  new  male,  employee,  was  making  suggestive  aematks  to  the  section  secAeta/iy. 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average,  or 
poor  performer? 

The  supeavisoA.  ovenheaAd  the  Aemanks  just  once  and  quickly  took  the  employee 
aside  and  lead  him  the  nJjot  act. 


3.  What  was  the  outcome  of  this  incident? 

The  new  employee  stopped  making  suggestive  Aemanks  to  the  secnetasiy.  The 
secnetany  neve a.  did  £ile  a  complaXnt. 


4.  In  what  supervisor  effectiveness  category  would  you  say  this  incident  falls? 

Resolving  problems  among  employees. 


5.  Circle  the  number  below  that  best 

reflects 

the  correct  effectiveness  level 

this  example. 

12  3  4 

5 

6  (T) 

8  9 

extreme 1 y  i nef f ec t i ve 

about 

effective 

extremely 

Ineffective 

average 

effective 
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Evaluation  of  tha  supervisor  Performance  Factors 

(Form  A) 


In  looking  over  the  preliminary  set  of  supervisor  performance  factors, 
yoc  may  have  felt  that  some  of  the  factors  overlapped  with  others  or  weren't 
too  important  in  arriving  at  an  overall  measure  of  supervisor  effectiveness. 
Later  in  this  research  project,  when  we  obtain  performance  ratings  oi< 
hundreds  of  individuals,  we  want  to  be  able  to  get  the  most  comprehensive 
overall  measure  of  effectiveness  that  we  can.  Yet  we  do  not  wish  to  burden 
the  raters  with  having  to  make  evaluations  on  a  large  number  of  performance 
factors.  We  would  like  to  obtain  the  judgments  of  a  representative  sample 
of  Army  civilian  supervisors  concerning  which  15  factors  would  produce  the 
best  measure  of  supervisor  effectiveness  when  combined  into  an  overall 
composite  index.  What  we  are  aiming  for  ultimately  is  a  set  of  scales  that 
efficiently  and  comprehensively  covers  the  different  factors  that  comprise 
Army  civilian  supervisor  effectiveness. 

If  you  feel  that  we  have  not  listed  one  or  more  key  factors  or 
dimensions  of  supervisor  performance,  please  write  a  brief  description  of 
the  f actor (s)  on  the  space  provided  on  the  bottom  of  the  form. 
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Evaluation  of  the  Supervisor  Performance  Factors 

(Form  A) 

Please  check  the  15  (no  more,  no  less)  factors  that  when  combined  would  provide  the 
most  comprehensive  overall  measure  of  supervisor  performance. 

Preliminary  Performance  Factor 

1.  Planning,  organizing,  and  assigning  personnel  and  other  resources  0 

2.  Coordinating  unit  activities  and  progress  with  other  units  and 

organizations  0 

3.  Displaying  effort,  conscientiousness,  and  responsibility  0 

4.  Motivating,  encouraging,  rewarding,  and  disciplining  subordinates  0 

5.  Maintaining  security  of  sensitive  or  classified  information  0 

6.  Recommending  the  selection,  promotion,  and  reassignment  of  subordinates  0 

7.  Performing  well  under  stress  and  adverse  conditions  0 

8.  Ensuring  quality,  quantity,  and  timeliness  of  unit  output  0 

9.  Identifying  problems  and  removing  work-related  obstacles  0 

10.  Monitoring,  reviewing,  and  evaluating  performance  of  subordinates  0 

11.  Giving  subordinates  advice/counsel  on  how  to  improve  performance  0 

12.  Displaying  technical  knowledge  and  skill  0  . 

13.  Hearing,  resolving,  and  referring  subordinate  complaints  and  grievances  0 

14.  Developing  own  job  and  supervisory  skills  0 

15.  Exercising  initiative  and  independent  judgment  0 

16.  Maintaining  a  positive  work  environment  for  subordinates  0 

17.  Following  regulations  and  administrative  guidelines  for  reports, 

records,  personnel  actions,  etc.  0 

18.  Identifying  and  providing  for  the  training  needs  of  subordinates  0 

19.  Orally  communicating  work  status  and  progress  to  superiors, 

subordinates,  and  others  0 

20.  Communicating  in  writing  work  status  and  progress  to  superiors, 

subordinates,  and  others  0 

21.  Displaying  honesty,  integrity,  and  a  sense  of  loyalty  0 

22.  Maintaining  unit  equipment  and  supplies  0 

23.  Maintaining  employee  safety  0 

24.  Counseling  employees  on  their  career  development  0 

25.  Delegating  work  when  appropriate  0 

Additional  Factors 
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Evaluation  of  tho  Supervisor  Performance  Factors 

(Form  B) 

It  is  often  easier  to  rate  individuals  on  some  aspects  of  performance 
than  on  others.  This  may  happen  because  some  behaviors  are  not  often  seen, 
are  difficult  to  interpret  clearly,  or  for  a  number  of  other  reasons.  We 
would  like  to  know  which  of  the  supervisor  performance  factors  in  your 
opinion  would  be  the  most  difficult  to  rate  reliably  and  validly.  Please 
use  Form  B  to  Indicate  which  factors  would  probably  give  you  the  most 
difficulty  in  rating  a  first  line  supervisor,  and  what  you  feel  the  source 
of  the  difficulty  would  be. 
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Evaluation  of  the  Supervisor  Performance  Factors 

(Form  B) 

Please  check  the  performance  factors  that  you  believe  would  be  post  difficult  to 
complete  in  rating  first  line  civilian  supervisors  and  indicate  the  likely  source  of 
the  difficulty. 


Source  of 

Preliminary  Performance  Factor  Difficulty 


1.  Planninq,  organizing,  and  assigning  personnel  and 

other  resources  0 

2.  Coordinating  unit  activities  and  progress  with  othet 

units  and  organizations  0 

3.  Displaying  effort,  conscientiousness,  and 

responsibility  0 

4.  Motivating,  encouraging,  rewarding,  and  disciplining 

subordinates  0 

5.  Maintaining  security  of  sensitive  or  classified 

information  0 

6.  Recommending  the  selection,  promotion,  and 

reassignment  of  subordinates  0 

7.  Performing  well  under  stress  and  adverse  conditions  0 

8.  Ensuring  quality,  quantity,  and  timeliness  of 

unit  output  0 

9.  Identifying  problems  and  removing  work-related 

obstacles  0 

10.  Monitoring,  reviewing,  and  evaluating  performance  of 

subordinates  0 

11.  Giving  subordinates  advice/counsel  on  how-to  improve 

performance  0 

12.  Displaying  technical  knowledge  and  skill  0 

13.  Hearing,  resolving,  and  referring  subordinate 

complaints  and  grievances  0 

14.  Developing  own  job  and  supervisory  skills  0 

15.  Exercising  initiative  and  independent  judgment  0 

16.  Maintaining  a  positive  work  environment  for 

subordinates  0 

17.  Following  regulations  and  administrative  guidelines 

for  reports,  records,  personnel  actions,  etc.  0 

18.  Identifying  and  providing  for  the  training  needs  of 

subordinates  0 

19.  Orally  communicating  work  status  and  progress  to 

superiors,  subordinates,  and  others  0 

20.  Communicating  in  writing  work  status  and  progress  to 

superiors,  subordinates,  and  others  0 

21.  Displaying  honesty,  integrity,  and  a  sense  of  loyalty 

22.  Maintaining  unit  equipment  and  supplies  0 

23.  Maintaining  employee  safety  0 

24.  Counseling  employees  on  their  career  development  0 

25.  Delegating  work  when  appropriate  0 
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Evaluation  of  tha  Supervisor  Performance  Factors 

(Fona  C) 


Some  of  the  supervisor  performance  factors  may  be  more  applicable  to 
supervisors  working  in  certain  jobs  than  in  others.  In  forming  an  ove.'*ll 
composite  measure  of  individual  effectiveness  for  supervisors  in  different 
Army  civilian  jobs,  some  rating  dimensions  may  really  be  inappropriate  or 
nonapplicable  for  measuring  performance  in  certain  kinds  of  jobs. 

Please  examine  the  list  of  performance  factors  on  Form  C  and  indicate 
for  which  types  of  jobs,  if  any,  some  of  the  factors  might  best  be  dropped 
when  forming  an  overall  measure  of  performance  for  supervisors  working  in 
those  jobs. 
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Supervisor  Perfomance  Factor  Evaluation 
(Form  C) 


Please  record  next  to  the  performance  factor  any  types  of  Army  civilian  jobs  for  which 
it  would  be  inappropriate  to  use  the  factor  in  measuring  supervisor  effectiveness. 


Preliminary  Performance  Factor 


1.  Planning,  organizing,  and  assigning  personnel  and 

other  resources  0 

2.  Coordinating  unit  activities  and  progress  with  other 

units  and  organizations  0 

3.  Displaying  effort,  conscientiousness,  and 

responsibility  0 

4.  Motivating,  encouraging,  rewarding,  and  disciplining 

subordinates  0 

5.  Maintaining  security  of  sensitive  or  classified 

Information  0 

6.  Recommending  the  selection,  promotion,  and 

reassignment  of  subordinates  0 

7.  Performing  well  under  stress  and  adverse  conditions  0 

8.  Ensuring  quality,  quantity,  and  timeliness  of 

unit  output  0 

9.  Identifying  problems  and  removing  work-related 

obstacles  0 

10.  Monitoring,  reviewing,  and  evaluating  performance  of 

subordinates  0 

11.  Giving  subordinates  advice/counsel  on  how  to  improve 

performance  0 

12.  Displaying  technical  knowledge  and  skill  0 

13.  Hearing,  resolving,  and  referring  subordinate 

complaints  and  grievances  0 

14.  Developing  own  job  and  supervisory  skills  0 

15.  Exercising  initiative  and  independent  judgment  0 

16.  Maintaining  a  positive  work  environment  for 

subordinates  0 

17.  Following  regulations  and  administrative  guidelines 

for  reports,  records,  personnel  actions,  etc.  0 

18.  Identifying  and  providing  for  the  training  needs  of 

subordinates  0 

19.  Orally  communicating  work  status  and  progress  to 

superiors,  subordinates,  and  others  0 

20.  Communicating  in  writing  work  status  and  progress  to 

superiors,  subordinates,  and  others  0 

21.  Displaying  honesty,  integrity,  and  a  sense  of  loyalty 

22.  Maintaining  unit  equipment  and  supplies  0 

23.  Maintaining  employee  safety  0 

24.  Counseling  employees  on  their  career  development  0 

25.  Delegating  work  when  appropriate  0 


Nonapplicable 
Army  Civilian  Jobs 
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Evaluation  of  th«  Supervisor  Performance  Factors 

(Form  0) 


Some  of  the  supervisor  performance  factors  you  have  been  evaluating  may 
have  seemed  to  you  to  be  essentially  measuring  the  same  basic  factor.  That 
is,  two  or  three  or  more  of  the  factors  may  have  seemed  to  be  overlapping 
each  other  to  the  extent  that  anyone  who  was  rated  high  (or  low)  on  one 
factor  would  almost  certainly  be  rated  high  (or  low)  on  the  other  factor(s). 
Unless  such  redundancy  is  eliminated,  the  overall  composite  measure  of 
Individual  supervisor  effectiveness  may  give  too  high  a  weight  to  these 
highly  similar  factors. 

Please  examine  the  list  of  performance  factors  on  Form  D  and  indicate 
which  factors,  if  any,  are  so  highly  similar  that  they  might  best  be 
combined  into  the  same  basic  factor  when  forming  an  overall  measure  of 
performance  for  supervisors. 
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Supervisor  Performance  Factor  Evaluation 
(Form  D) 


Please  record  next  to  each  performance  factor  the  Identification  number  of  other 
factors,  if  any,  that  are  essentially  measuring  the  same  thing  as  the  performance 
factor. 


Preliminary  Performance  Factor 


1.  Planning,  organizing,  and  assigning  personnel  and 

other  resources  0 

2.  Coordinating  unit  activities  and  progress  with  other 

units  and  organizations  '  0 

3.  Displaying  effort,  conscientiousness,  and 

responsibility  0 

4.  Motivating,  encouraging,  rewarding,  and  disciplining 

subordinates  0 

5.  Maintaining  security  of  sensitive  or  classified 

information  0 

6.  Recommending  the  selection,  promotion,  and 

reassignment  of  subordinates  0 

7.  Performing  well  under  stress  and  adverse  conditions  0 

8.  Ensuring  quality,  quantity,  and  timeliness  of 

unit  output  0 

9.  Identifying  problems  and  removing  work-related 

obstacles  0 

10.  Monitoring,  reviewing,  and  evaluating  performance  of 

subordinates  0 

11.  Giving  subordinates  advice/counsel  on  how  to  improve 

performance  0 

12.  Displaying  technical  knowledge  and  skill  0 

13.  Hearing,  resolving,  and  referring  subordinate 

complaints  and  grievances  0 

14.  Developing  own  job  and  supervisory  skills  0 

15.  Exercising  initiative  and  independent  judgment  0 

16.  Maintaining  a  positive  work  environment  for 

subordinates  0 

17.  Following  regulations  and  administrative  guidelines 

for  reports,  records,  personnel  actions,  etc.  0 

18.  Identifying  and  providing  for  the  training  needs  of 

subordinates  0 

19.  Orally  communicating  work  status  and  progress  to 

superiors,  subordinates,  and  others  0 

20.  Communicating  in  writing  work  status  and  progress  to 

superiors,  subordinates,  and  other  0 

21.  Displaying  honesty,  integrity,  and  a  sense  of  loyalty  0 

22.  Maintaining  unit  equipment  and  supplies  0 

23.  Maintaining  employee  safety  0 

24.  Counseling  employees  on  their  career  development  0 

25.  Delegating  work  when  appropriate  0 


ID  No.  of  Highly 
Similar  Factor(s) 
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DISCUSSION  QUESTIONS 


1.  Does  anyone  have  any  questions  about  what  we  did  today,  or  any  problems 
or  issues  about  measuring  first-line  supervisor  performance  that  they 
would  like  to  bring  up? 


2.  Did  anyone  write  down  any  additional  first-line  performance  factors  that 
would  be  important  to  rate?  Are  there  any  additional  ones  that  should 
be  considered? 


3.  How  long  should  a  rater  (a  higher  level  supervisor)  know  a  first-line 
supervisor  to  be  able  to  validly  rate  him/her  on  these  types  of 
performance  factors? 


4.  Should  we  try  to  get  peer  ratings  of  first-line  supervisors  as  well  as 
the  ratings  of  higher  level  supervisors  to  get  a  better  overall  measure 
of  first-line  supervisor  performance? 


5.  Are  there  any  other  types  of  measures  that  we  could  get  which  would  be 
useful  in  the  measurement  of  first- line  supervisor  performance? 


APPENDIX  B 

MATERIALS  USED  IN  RETRANSLATION  WORKSHOPS 
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AGENDA 


First  Line  Supervisor  Job  Performance  Workshop 


Time1 

TodIc 

0830  -  0845 

Project  overview.  Description  of 
workshop  activities. 

0845  -  0900 

The  preliminary  set  of  performance 
categories 

0900  -  1130 

Classify  critical  incidents  into 
performance  categories 

1130  -  1200 

Evaluation  of  the  preliminary  set  of 
performance  categories 

1  Time  periods  depended  upon  workshop  starting  time. 
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ORIENTATION  TO  CIVPERS  PROJECT 


Most  managers  would  agree  that  the  civilian  first- line  supervisor 
plays  a  critical  role  in  the  Army  and  that  the  selection  of  the  best  first- 
line  supervisor  candidates  is  an  essential  component  of  Army  readiness. 
However,  officials  responsible  for  selection  have  difficulty  estimating  the 
supervisory  potential  of  candidates  because  few  applicants  have  ever  worked 
as  supervisors.  As  a  result,  in  many  cases  the  overriding  factor  in  a 
candidate's  selection  is  technical  competence  rather  than  supervisory 
potential.  A  research  effort  has  been  recently  initiated  by  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  to  evaluate  the 
supervisory  aspects  of  the  job  of  civilian  first- line  supervisors  in  the 
Army  and  to  develop  a  set  of  supervisory  selection  measures  which  can  more 
accurately  assess  supervisory  potential. 

To  help  accomplish  this  research,  ARI  has  retained  a  team  of 
contractors  led  by  the  Human  Resources  Research  Organization  (HumRRO).  The 
research  team  is  composed  of  the  following  organizations:  HumRRO,  Caliber 
Associates,  Systems  Research  and  Applications  Corporation  (SRA),  and  Booz- 
Allen  Hamilton.  Our  research  team  is  known  for  developing  products  that 
are  immediately  useful  to  sponsors. 

Today,  we  will  first  review  what  types  of  first-line  supervisory 
selection  instruments  are  going  to  be  developed  for  Civilian  personnel, 
then,  we  would  like  your  help  in  providing  us  with  crucial  information  that 
will  aid  us  in  the  development  of  these  Civilian  First-line  supervisory 
selection  instruments. 
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CANDIDATE  SELECTION 
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First  Line  Supervisors 

•  Structured  Interview  Guide 

•  Scoreable  In-Basket  Exercise 


CANDIDATE  SELECTION 


Candidate  selection  work  addresses  the  need  to  select  first-line 
supervisors  with  strong  supervisory  potential.  Recognizing  that  over 
26,000  first-line  supervisors  currently  serve  in  the  DA  civilian 
workforce  and  that  over  3000  first-line  supervisor  promotions  are  made 
annually,  we  made  development  of  valid  selection  ;  rocedures  a  top 
priority. 

Our  research  and  development  efforts  indicate  that  the  neeu  for 
valid  selection  procedures  will  be  best  met  by  at  least  two  procedures: 

•  A  Structured  Interview  Guide  -  designed  to  specify  crucial 
topics  to  address  and  assess  in  the  personal  interview  setting. 

•  A  Scoreable  In-basket  Exercise  -  a  judgment  and  decision-making 
exercise  with  scoring  criteria  to  reduce  administration  time. 
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WORKSHOP  OVERVIEW 


In  order  to  determine  whether  scores  on  the  selection  instruments  are 
re luted  to  on-the-job  performance,  it  is  vital  to  determine  whether  first- 
l4,ne  supervisors  who  do  well  on  the  selection  instruments  also,  in  fact,  do 
well  o'-i  the  job.  Conversely,  do  low  scorers  on  the  selection  tests  perform 
less  effectively  in  supervisory  jobs?  To  answer  these  questions,  we  need  to 
obtain  an  accurate  picture  of  the  job  performance  of  individual  first-line 
supervisors. 

We  are  going  to  work  together  to  develop  a  state-of-the-art  rating 
form.  This  form  should  help  raters  (for  example,  second-line  supervisors  or 
fellow  first-line  supervisors)  make  accurate  judgments  of  the  actual 
performance  effectiveness  of  supervisors  they  either  supervise  or  work  with. 
The  first  steps  in  developing  these  rating  scales  were  accomplished  recently 
when  groups  of  supervisors  like  yourselves  wrote  a  series  of  critical 
incidents  describing  what  some  first-line  supervisors  did  or  failed  to  do  in 
work  situations  that  made  them  effective  or  ineffective  performers  in  those 
situations.  These  incidents  were  then  used  to  identify  a  preliminary  set  of 
performance  categor i es . 

Now,  in  the  third  major  step,  it  is  necessary  to  determine  whether 
supervisors  like  yourselves  can  reliably  categorize  the  behavior  described 
in  the  incidents  to  these  performance  categories.  It  is  also  necessary  to 
determine  whether  you  agree  on  the  effectiveness  level  of  the  supervisor 
performance. 
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By  developing  performance  scales  based  on  actual  behavioral  examples, 
we  should  be  able  to  obtain  ratings  which  emphasize  objective  observation 
more  than  subjective  evaluation.  Furthermore,  the  scales  should  reflect 
performance  requirements  regarded  as  important  by  those  knowledgeable  about 
Army  civilian  first- line  supervisor  jobs. 
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Examples  of  behavior-based  rating  scales 


tow  urn* 


How  effective  is  the  supervisor  in  showing  consideration  toward  and 

providing  help  to  subordinates? 


Ignores  subordinates' 
feelings  and  refuses  or 
otherwise  fails  to  help 
with  work-related  or 
personal  problems;  often 
neglects  to  pass  on 
important  Information  to 
subordinates. 


1  2 


In  most  situations, 
helps  subordinates  with 
their  work-related/ 
personal  problems; 
usually  informs  unit 
members  about  what  they 
need  to  know. 


3  4  5 


Takes  extra  steps  to 
help  subordinates  with 
work-related  and 
personal  problems; 
always  informs  unit 
members  concerning 
information  they  should 
know  about. 


6  7 


_ SELF-PEVELQPMENT _ 

How  effective  is  the  supervisor  in  developing  own  job 
and  leadership  skills? 


Makes  little  or  no 
effort  to  improve 
technica 1 / leaders h  i  p 
skills; 

refuses  or  otherwise 
fails  to  participate 
in  relevant  training 
courses  or  career 
growth  opportunities. 


1  2 


Studies,  practices,  or 
participates  in 
course/training  when 
required  to  do  so  or 
when  encouraged  to  work 
on  problem  areas. 


3  4  5 


Takes  courses,  studies, 
and  wcrks  hard  during 
non- work  hours  to 
improve  technical  and 
leadership  skills; 
actively  seeks 
additional  job  duties/ 
responsibilities  to 
improve  job  skills  and 
increase  chances 
of  promotion. 


6  7 


4 
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Workshop  Activities 


« 


» 


•  Examine  preliminary  set  of  performance  categories 

•  Classify  critical  incidents  into  performance  categories 

•  Evaluate  effectiveness  of  the  supervisor  performance 
described  in  the  critical  incidents 

•  Evaluate  the  preliminary  set  of  performance  categories 
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First-Line  Supervisor  Performance  Categories 


A.  Enforcing  Standards  of  Behavior  and  Adherence  to  Policies/Regulations 

•  Sets  clear  standards  of  acceptable  behavior  on  the  job.  Insists 
employees  meet  these  standards  and  follow  applicable  policy 
directives,  rules  and  regulations.  Takes  appropriate  unofficial  and 
official  disciplinary  actions  to  correct  failings. 

versus 

•  Ignores  such  behaviors  as  chronic  tardiness,  long  lunches,  improper 
dress,  excessive  phone  calls.  Is  careless  concerning  security 
matters,  record  keeping,  property  disposition,  travel  orders,  and  the 
like.  Over  or  under  reacts  in  administering  discipline,  failing  to 
obtain  improvement. 


B.  Maintaining  Employees  Safety,  Health  and  Physical  Hell  Being 

•  Takes  active  steps  to  minimize  hazards  in  the  work  environment. 

Calls  attention  of  subordinates  and  proper  authorities  to  safety  and 
health  problems.  Ensures  that  unit  work  environment  is  clean, 
adequately  heated  and  lighted,  free  from  clutter,  noisy  distractions, 
cigarette  smoke,  and  the  like. 

versus 

t  Ignores  safety  and  health  hazards.  Fails  to  enforce  safety  and 
health  related  rules,  e.g.,  smoking  restrictions.  Shows  no  concern 
for  physical  well  being  of  subordinates. 


C.  Completing  Formal  Fmployee  Performance  Appraisals 

•  Completes  performance  appraisals  for  annual  reviews  and  promotions  on 
time,  and  constructively.  Discusses  strength  and  weaknesses  of 
employees.  Evaluates  performance  objectively  and  fairly. 

versus 

•  Delays  completing  performance  appraisals;  fails  to  indicate  how 
performance  could  be  improved.  Plays  favorites,  gives  high  (or  low) 
ratings  to  individuals  who  don't  deserve  them. 
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D.  Resolving  Enployee  Conflicts 

•  Actively  intercedes  to  reduce  conflicts  among  employees.  Listens  to 
each  side  attentively  and  tries  to  promote  positive  interactions. 
Takes  strong  action  when  necessary  to  reduce  tensions  and  conflicts. 

versus 

•  Ignores  conflicts  among  employees,  lets  them  get  out  of  hand;  jumps 
in  thoughtlessly,  without  ascertaining  facts.  Shows  bias  in  handling 
disputes. 


E.  Providing  Personal  and  Career  Counseling 

•  Provides  confidential  advice  and  help  to  subordinates  whose  personal 
problems  are  interfering  with  performance.  Advises  subordinates  on 
job  growth  and  career  possibilities  and  how  to  overcome  personal 
obstacles  to  job  success. 

versus 

•  Ignores  personal  problems  of  subordinates  or  makes  them  worse  through 
callousness  and  insensitivity.  Discourages  career  and  personal 
growth  of  subordinates. 


F.  Communicating  with  Subordinates 

•  Keeps  subordinates  informed  about  impending  personnel  and  work  load 
changes,  installation  activities,  new  policies,  and  other  matters  of 
concern  and  interest  to  them.  Regularly  holds  meetings  with 
subordinates  to  discuss  such  items.  Encourages  employee  suggestions 
and  comments. 

versus 

•  Fails  to  tell  employees  about  upcoming  changes  that  may  impact  their 
jobs  and  work  environment.  Keeps  such  information  to  self,  sometimes 
to  detriment  of  employees'  job  performance  and  satisfaction. 
Discourages  exchange  of  ideas  and  opinions  on  job  related  matters. 
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G.  Assuring  Technical  Competence  of  Subordinates 

§  Makes  sure  that  subordinates  understand  how  to  do  their  tasks 
properly  by  providing  relevant  and  timely  technical  guidance  and 
feedback.  Closely  monitors  quality  and  timeliness  of  subordinates' 
products  and  provides  constructive  suggestions  for  improving 
performance.  Provides  any  needed  on-the-job  training  or  ensures  that 
it  is  obtained  elsewhere. 

versus 

•  Doesn't  provide  the  technical  guidance  or  training  necessary  for 
subordinates  to  do  high  quality  work.  Fails  to  monitor  subordinates' 
task  accomplishment  and  to  ensure  timely,  accurate,  or  complete 

•  reports,  quality  products. 


H.  Showing  Initiative,  Extra  Effort,  and  Composure  Under  Stress 

•  Reacts  to  job  demands- and  stressful  conditions  with  initiative,  extra 
effort,  and  composure.  Voluntarily  tackles  new  tasks.  Swiftly  acts 
to  remove  unexpected  barriers  to  job  success.  Efficiently 
accomplishes  complex  tasks  under  extreme  time  pressures. 

versus 

•  Seems  to  fall  apart  under  pressure,  not  able  to  rise  to  the  occasion. 
Doesn't  do  more  than  what's  minimally  necessary  to  maintain  position. 
Fails  to  put  in  extra  effort  when  needed. 


I.  Showing  Integrity  and  Honesty 

•  Is  honest  and  straightforward  in  dealings  with  others.  Follows  same 
rules  set  for  subordinates.  Gives  subordinates  credit  when  credit  is 
due.  Sets  a  role  model  for  subordinates  to  follow. 

versus 

•  Exploits  position  for  personal  gain.  Takes  sole  credit  for 
subordinates'  work.  Lies  to  or  misleads  subordinates.  Engages  in 
dishonest  acts  such  as  pilfering,  falsifying  leave  records,  charging 
time  not  worked. 
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J.  Maintaining  Employee  Morale  and  Motivation 

•  Takes  steps  to  ensure  that  employee  moral  is  high  and  that  employees 
are  well  motivated.  For  example,  provides  awards  fairly,  pays 
individualized  attention  to  employees,  arranges  special  ceremonies 
and  events,  encourages  team  spirit  and  high  unit  performance. 

versus 

•  Makes  employees  feel  generally  unappreciated.  Fails  to  consider 
special  needs  of  subordinates  or  back  them  up  when  they  need  support. 
Frequently  undercuts  subordinates. 


K.  Assuring  Unit  Functioning 

•  Carefully  plans  and  schedules  unit  work.  Reorganizes  unit  activities 
and  staff  responsibilities  and  annual  leave  to  meet  changes  in  work 
loads  and  deadlines.  Assigns  employees  to  achieve  fair,  challenging 
work  load  distribution.  Makes  sure  subordinates  are  in  appropriate 
grades.  Protects  unit  from  outside  interference,  staff  reductions, 
mission  loss.  Develops  own  professional  skills  as  a  manager. 

versus 

•  Plans  and  schedules  Inadequately  so  that  backlogs,  bottlenecks, 
machine  failures,  supply  and  staff  shortages  occur.  Delegates  work 
tasks  unequally;  assigns  work  without  considering  duties  and 
capabilities  of  staff.  Allows  major  disruption  of  unit  functioning 
to  occur.  Makes  no  attempt  to  grow  as  a  supervisor  or  manager. 


L.  Communicating,  Coordinating  With  Supervisors  and  Other  Units 

•  Communicates  effectively  with  supervisors  and  other  units.  Keeps 
them  informed  about  unit  activities,  product  delivery,  personnel 
availability,  and  problems.  Handles  disagreements  with  tact, 
diplomacy,  and  persuasiveness.  Coordinates  inter-unit  activities  to 
improve  mutual  productivity. 

versus 

•  Gets  into  no-win  struggles  with  supervisors  and  other  units.  Is 
rude,  abrasive,  and  disrespectful  to  them.  Hides  problems  until  it's 
too  late  to  effectively  solve  them.  Fosters  "them"  versus  "us" 
feelings. 
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FIRST  EXAMPLE 


SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  Mere  the  circumstances  leading  up  to  the  incident? 

An  employee  was  coming  in  late  and  staying  late  to  work  overtime  routinely, 
thereby  falsely  crediting  50  hours  per  week  worked. 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average, 
or  poor  performer? 

The  supervisor  did  nothing  and  claimed  he  wasn't  aware  of  the  problem. 


3.  What  was  the  outcome  of  this  incident? 

The  supervisor  was  spoken  to  and  authorization  of  overtime  for  that  employee 
was  not  approved. 


4.  Supervisor  performance  category: 


5.  Circle  the  number  below  that  best  reflects  the  correct  effectiveness  level 
for  this  example. 


1  2 

3  4 

5 

6  7  8 

9 

extremely 

ineffective 

about 

effective 

extremely 

ineffective 

average 

effective 
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SECOND  EXAMPLE 


SUPERVISOR  PERFORMANCE  INCIDENT  FORM 


1.  What  were  the  circuastances  leading  up  to  the  incident? 

Performance  appraisal  was  prepared  for  a  new  employee  with  limited  experience 
and  some  shortcomings.  Also,  some  positive  aspects  and  potential. 


2.  What  did  the  individual  do  that  made  you  feel  he  or  she  was  a  good,  average, 
or  poor  performer? 

The  rating  reflected  all  the  strong  and  weak  points  of  tie  employee  and  were 
right  on  the  mark.  The  employee  was  made  aware  of  her  “‘•sessment  with  each 
point  being  discussed  including  her  potential. 


3.  What  was  the  outcome  of  this  incident? 

The  employee  showed  considerable  improvement  after  the  rating,  and,  if  this 
continues,  will  warrant  a  higher  rating  with  a  strong  possibility  for 
promotion. 


4.  Supervisor  performance  category: 


5.  Circle  the  number  below  that  best  reflects  the  correct  effectiveness  level 
for  this  example. 


1  2 

3  4 

5 

6  7  8 

9 

extremely 

ineffective 

ineffective 

about 

average 

effective 

extremely 

effective 
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First-Line  Supervisor  Performance  Categories 


A.  Enforcing  Standards  of  Behavior  and  Adherence  to  Policies/Regulations 

B.  Maintaining  Employee  Safety,  Health  and  Physical  Well  Being 

C.  Completing  Formal  Employee  Performance  Appraisals 

D.  Resolving  Employee  Conflicts 

E.  Providing  Personal  and  Career  Counseling 

F.  Communicating  with  Subordinates 

G.  Assuring  Technical  Competence  of  Subordinates 

H.  Showing  Initiative,  Extra  Effort,  and  Composure  Under  Stress 

I.  Showing  Integrity  and  Honesty 

J.  Maintaining  Employee  Morale  and  Motivation 

K.  Assuring  Unit  Functioning 

L.  Communicating,  Coordinating  with  Supervisors  and  Other  Units 
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THE  EVALUATION  OF  THE  PRELIMINARY  SET  OF  PERFORMANCE  CATEGORIES 


In  order  to  further  the  development  of  a  comprehensive  set  of  behavior- 
based  rating  scales,  we  would  like  you  to  evaluate  the  preliminary  set  of  12 
performance  categories.  Four  different  aspects  or  dimensions  will  be 
considered  in  the  evaluation: 

(1)  which  categories  would  best  be  used  to  form  a  comprehensive 
overall  measure  of  first- line  supervisor  performance; 

(2)  which  performance  categories  would  present  the  most  difficulty  in 
rating  first- line  supervisors; 

(3)  which  performance  categories  would  not  be  applicable  to  first-line 
supervisors  in  certain  types  of  jobs;  and 

(4)  which  performance  categories  overlap  other  categories  so  much  that 
they  are  essentially  measuring  the  same  thing. 

The  rationale  for  each  of  these  evaluations  is  presented  more  fully  on 
the  following  pages.  A  separate  form  is  provided  for  each  evaluation. 

Thank  you  for  your  cooperation. 
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Evaluation  of  the  Supervisor  Performance  Categories 

(Form  A) 


In  looking  over  the  preliminary  set  of  supervisor  performance 
categories,  you  may  have  felt  that  some  of  the  categories  overlapped  with 
others  or  weren't  too  important  in  arriving  at  an  overall  measure  of 
supervisor  effectiveness.  Later  in  this  research  project,  when  we  obtain 
performance  ratings  on  hundreds  of  individuals,  we  want  to  be  able  to  get 
the  most  comprehensive  overall  measure  of  effectiveness  that  we  can.  Yet  we 
do  not  wish  to  burden  the  raters  with  having  to  make  evaluations  on  a  large 
number  of  performance  scales.  We  would  like  to  obtain  the  judgments  of  a 
representative  sample  of  Army  civilian  supervisors  concerning  which  eight 
categories  would  produce  the  best  mearure  of  supervisor  effectiveness  when 
combined  into  an  overall  composite  index.  What  we  are  aiming  for  ultimately 
Is  a  set  of  scales  that  efficiently  and  comprehensively  covers  the  different 
factors  that  comprise  Army  civilian  supervisor  effectiveness. 

If  you  feel  that  we  have  not  listed  one  or  more  key  categories  or 
dimensions  of  supervisor  performance,  please  write  a  brief  description  o * 
the  categories  on  the  space  provided  on  the  bottom  of  the  form. 
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Evaluation  of  the  Supervisor  Performance  Categories 

(Fora  A) 


Please  check  the  8  (no  more,  no  less)  categories  that  when  combined  would 
provide  the  most  comprehensive  overall  measure  of  supervisor  performance. 


Preliminary  Performance  Category 

A.  Enforcing  Standards  of  Behavior  and 

Adherence  to  Policies/Rogulations  0 

B.  Maintaining  Employee  Safety,  Health  and 

Physical  Well  Being  0 

C.  Completing  Formal  Employee  Performance 

Appraisals  0 

D.  Resolving  Employee  Conflicts  0 

E.  Providing  Personal  and  Career  Counseling  0 

F.  Communicating  with  Subordinates  0 

G.  Assuring  Technical  Competence  of 

Subordinates  0 

H.  Showing  Initiative,  Extra  Effort,  and 

Composure  Under  Stress  0 

I.  Showing  Integrity  and  Honesty  0 

J.  Maintaining  Employee  Morale  and  Motivation  0 

K.  Assuring  Unit  Functioning  0 

L.  Communicating,  Coordinating  with 

Supervisors  and  Other  Units  0 


Additional  Factors 
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Evaluation  of  the  Supervisor  Performance  Categories 

(Form  B) 


It  is  often  easier  to  rate  individuals  on  some  aspects  of  performance 
than  on  others.  This  may  happen  because  some  behaviors  are  not  often  seen, 
are  difficult  to  interpret  clearly,  or  for  a  number  of  other  reasons.  We 
would  like  to  know  which  of  the  supervisor  performance  categories  in  your 
opinion  would  be  the  most  difficult  to  rate  reliably  end  validly.  Please 
use  Form  B  to  indicate  which  categories  would  probably  give  you  the  most 
difficulty  in  rating  a  first- line  supervisor,  and  what  you  feel  the  source 
of  the  difficulty  would  be. 
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Evaluation  of  the  Supervisor  Performance  Categories 

(Fora  B) 


Please  check  the  performance  categories  that  you  believe  would  be  most 
difficult  to  complete  in  rating  first- line  civilian  supervisors  and  indicate 
the  likely  source  of  the  difficulty. 


Preliminary  Performance  Category 

A.  Enforcing  Standards  of  Behavior  and 

Adherence  to  Policies/Regulations  0 

B.  Maintaining  Employee  Safety,  Health  and 

Physical  Well  Being  0 

C.  Completing  Formal  Employee  Performance 

Appraisals  0 

D.  Resolving  Employee  Conflicts  0 

E.  Providing  Personal  and  Career  Counseling  0 

F.  Communicating  with  Subordinates  0 

G.  Assuring  Technical  Competence  of 

Subordinates  0 

H.  Showing  Initiative,  Extra  Effort,  and 

Composure  Under  Stress  0 

I.  Showing  Integrity  and  Honesty  0 

J.  Maintaining  Employee  Morale  and  Motivation  0 

K.  Assuring  "nit  F-.tctioning  0 

L.  Communicating,  Coordinating  with 

Supervisors  and  Other  Units  0 


Source  of 
Difficulty 
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Evaluation  of  the  Supervisor  Performance  Categories 

(Fora  C) 

Some  of  the  supervisor  performance  categories  may  be  more  applicable  to 
supervisors  working  in  certain  jobs  than  in  others.  In  forming  an  overall 
composite  measure  of  individual  effectiveness  for  supervisors  in  different 
Army  civilian  jobs,  some  rating  dimensions  may  really  be  inappropriate  or 
nonapplicable  for  measuring  performance  in  certain  kinds  of  jobs. 

Please  examine  the  list  of  performance  categories  on  Form  C  and  indicate 
for  which  types  of  jobs,  if  any,  some  of  the  factors  might  best  be  dropped 
when  forming  an  overall  measure  of  performance  for  supervisors  working  in 
those  jobs. 
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Supervisor  Performance  Category  Evaluation 
(Form  C) 


Please  record  next  to  the  performance  category  any  types  of  Army  civilian 
jobs  for  which  it  would  be  inappropriate  to  use  the  category  in  measuring 
supervisor  effectiveness. 


Preliminary  Performance  Category 

A.  Enforcing  Standards  of  Behavior  and 
Adherence  to  Policies/Regulations 

B.  Maintaining  Employee  Safety,  Health  and 
Physical  Well  Being 

C.  Completing  Formal  Employee  Performance 
Appraisals 

D.  Resolving  Employee  Conflicts 

E.  Providing  Personal  and  Career  Counsiling 

F.  Communicating  with  Subordinates 

G.  Assuring  Technical  Competence  of 
Subordinates 


Nonappli cable 
Army  Civilian  Jobs 


0 

0 

0 

0 

0 

0 

0 


H.  Showing  Initiative,  Extra  Effort,  and 

Composure  Under  Stress  0 

I.  Shewing  Integrity  and  Honesty  0 

J.  Maintaining  Employee  Morale  and  Motivation  0 

K.  Assuring  Unit  Functioning  0 

L.  Communicating,  Coordinating  with 

Supervisors  and  Other  Units  0 
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Evaluation  of  the  Supervisor  Performance  Category 

(Form  D) 


Some  of  the  supervisor  performance  categories  you  have  been  evaluating 
may  have  seemed  to  you  to  be  essentially  measuring  the  same  basic  factor. 
That  is,  two  or  three  or  more  of  the  categories  may  have  seemed  to  be 
overlapping  each  other  to  the  extent  that  anyone  who  was  rated  high  (or  low) 
in  one  category  would  almost  certainly  be  rated  high  (or  low)  in  the  other 
category(s).  Unless  such  redundancy  is  eliminated,  the  overall  composite 
measure  of  individual  supervisor  effectiveness  may  give  too  high  a  weicliJ  to 
these  highly  similar  performance  categories. 

Please  examine  the  list  of  performance  categories  on  Form  D  and  indicate 
which  categories,  if  any,  are  so  highly  similar  that  they  might  best  be 
combined  into  the  same  basic  factor  when  forming  an  overall  measure  of 
performance  for  supervisors. 
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Supervisor  Performance  Category  Evaluation 
(Form  D) 


Please  record  next  to  each  performance  category  the  identification  number  of 
other  categories,  if  any,  that  are  essentially  measuring  the  same  thing  as 
the  performance  category. 


ID  Letter  of  Highly 


Preliminary  Performance  Cateaorv 

Similar  Cateaorv! si 

A. 

Enforcing  Standards  of  Behavior  and 

Adherence  to  Policies/Regulations 

0 

B. 

Maintaining  Employee  Safety,  Health  and 
Physical  Well  Being 

0 

C. 

Completing  Formal  Employee  Performance 
Appraisals 

0 

D. 

Resolving  Employee  Conflicts 

0 

E. 

Providing  Personal  and  Career  Counseling 

0 

F. 

Communicating  with  Subordinates 

0 

G. 

Assuring  Technical  Competence  of 

Subordinates 

0 

H. 

Showing  Initiative,  Extra  Effort,  and 
Composure  Under  Stress 

0 

I. 

Showing  Integrity  and  Honesty 

0 

J. 

Maintaining  Employee  Morale  and  Motivation 

0 

K. 

Assuring  Unit  Functioning 

0 

L. 

Communicating,  Coordinating  with 

Supervisors  and  Other  Units 

0 
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This  performance  rating  booklet  contains  a  preliminary  set  of  rating  scales  designed  to  assess  the  effectiveness  of  first  line 
supervisors. 
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’.hank  you  very  much  for  giving  this  your  careful  attention. 


ENFORCING  STANDARDS  OF  BEHAVIOR  AND  ADHERENCE  TO  POLICIES/REGULATIONS 
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First-line  supervisors 
you  are  rating. 


ASSURING  UNIT  FUNCTIONING 
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APPENDIX  C 

SUPERVISOR  RATING  SCALES  SELECTED  FOR  USE  IN  TRIAL  ADMINISTRATION 
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How  effective  is  this  supervisor  in  cowwunicating  with  subordinates? 
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L.  COMMUNICATING,  COORDINATING  WITH  SUPERVISORS  AND  OTHER  UNITS 
How  effective  is  this  supervisor  in  coaauni eating  and  coordinating  with  own  supervisors  and  other  units? 
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